What is cluster ?

What is cluster ?

Ans : A cluster is two or more interconnected computers that create a solution to provide higher availability, higher scalability or both. The advantage of clustering computers for high availability is seen if one of these computers fails, another computer in the cluster can then assume the workload of the failed computer. Users of the system see no interruption of access.

Clustering software : Red Hat Cluster suite
Platform : Red Hat Enterprise Linux 4,5,6
Storage : SCSI, SAN, NAS
Storage Protocols : iSCSI(pronounced "eye-scuzzy) /FCP
iSCSI => protcol to connect server to storage over IP network. Needs iSCSI initiator util will be on source or server. iSCSI target util will be on storage/target machine.
FCP => fibre channel protocol to connect server to storage over optical channel. Here needs HBA(host bus adapter like NIC) cards. It driver accesses this HBA and HBA communicates to SAN switch/storage controller.(Drivers like qla2xxx of qlogic company, lpfc(of emulex) etc)

Concepts: iSCSI is a protocol whereas SCSI is storage disk. consists using initiator(software+ hardware) and target. Initiator send packet to HBA. Target resides on storage like EqualLogic, NetApp filer, EMC NS-series or a HDS HNAS computer appliance. These attched with LUN to the drives. LUN is logical unit on storage treats as device or drive.

Storage System Connection Types :

a)active/active : all paths active all time
b)active‐passive : one path is active and other is backup
c)virtual port storage system.

Multipathing and Path Failover : When transferring data between the host server and storage, the SAN uses a multipathing technique where package "device-mapper-multipath" will have to be installed on server/node. The daemon "multipathd" periodically checks the connection paths to the storage. Multipathing allows you to have more than one physical path from the Server host to a LUN(treat is a device) on a storage system. If a path or any component along the path—HBA or NIC, cable, switch or switch port, or storage processor—fails, the server selects another of the available paths. The process of detecting a failed path and switching to another is called path failover.

Installation of Red Hat Cluster Suite on RHEL 5 :

1. Register system to RHN(Needs subscription with Red Het). Ignore if system is already registered :

---
#rhn_register
---

2. Use following command :

----
#yum groupinstall clustering cluster-storage
----

To separately install it do :

----
For Standard kernel :

#yum cman cman-kernel dlm dlm-kernel magma magma-plugins system-config-cluster rgmanager ccs fence modcluster --force

For SMP kernel :

#yum cman cman-kernel-smp dlm dlm-kernel-smp magma magma-plugins system-config-cluster rgmanager ccs fence modcluster --force
----

3. This steps should be followed on each nodes.

Configuring Red Hat Cluster Suite :

Configuration can be achieved in three ways like :
a) Using web interface(Conga tools) ie ricci and luci.
Conga — This is a comprehensive user interface for installing, configuring, and managing Red Hat clusters, computers, and storage attached to clusters and computers.

1. #yum install luci // Do on one machine(say A) to manage nodes. This machine may be outof clustered nodes.

2.Now initialize luci like

#luci_admin init

3. Install ricci on each nodes like :
#yum install ricci
4. Then access A(where luci has installed) like : http://IPof_A:port. Note that you'll get the url when you'll execute #luci_admin init in the above step.

b) Using "system-config-cluster" GUI interface. This is a user interface for configuring and managing a Red Hat cluster. Just use this command. Sometime it may not work if the server/node doesn't have GUI package like gnome or KDE.
c) Using "Command line tools" — This is a set of command line tools for configuring and managing a Red Hat cluster.

4. Different Clustered services (Ordered as per the manually starting queue): ccsd, cman, fence, rgmanager.
If you use LVM with GFS : ccsd, cman, fence, clvmd, gfs, rgmanager.

5. Configuration file (will be same on each nodes): /etc/cluster/cluster.conf, /etc/sysconfig/cluster. While you'll configure it using web interface, it'll be automatically copied on each nodes. Make sure you have enabled all the ports in firewall or disabled the firewall on all nodes as well as on luci node.

6. Now login into LUCI web interface and create a new cluster and give a name. Then in this lcuster add each nodes one by one. In this cluster add one fail over domain like httpd.(Make sure you have installed the httpd on each nodes where all the configuration files are same.). I shall describe it later and show you the result of real fail over testing.

Shared Disk configure(Disk size minimum 10MB is enough) : Why it is needed ?

AA) The shared partitions are used to hold cluster state information including "Cluster lock states", "Service states", "Configuration information". The shared disk may be on any node or on storage disk( will be connected to HBA, RAID controller(raid 1 ie mirror). This will be for shared disk(primary partition+shadow). Each minimum 10MB. Two raw devices on shared disk storage must be created for the primary shared partition and the shadow shared partition. Each shared partition must have a minimum size of 10 MB. The amount of data in a shared partition is constant; it does not increase or decrease over time. Periodically, each member writes the state of its services to shared storage. In addition, the shared partitions contain a version of the cluster configuration file. This ensures that each member has a common view of the cluster configuration. If the primary shared partition is corrupted, the cluster members read the information from the shadow (or backup) shared partition and simultaneously repair the primary partition. Data consistency is maintained through checksums, and any inconsistencies between the partitions are automatically corrected. If a member is unable to write to both shared partitions at start-up time, it is not allowed to join the cluster. In addition, if an active member can no longer write to both shared partitions, the member removes itself from the cluster by rebooting (and may be remotely power cycled by a healthy member).

BB) The following are shared partition requirements:
a)Both partitions must have a minimum size of 10 MB.
b)Shared partitions must be raw devices since file cache won't be there. They cannot contain file systems.
c)Shared partitions can be used only for cluster state and configuration information.

CC) Following are recommended guidelines for configuring the shared partitions(By Red Hat):

a)It is strongly recommended to set up a RAID subsystem for shared storage, and use RAID 1 (mirroring) to make the logical unit that contains the shared partitions highly available. Optionally, parity RAID can be used for high availability. Do not use RAID 0 (striping) alone for shared partitions.
b)Place both shared partitions on the same RAID set, or on the same disk if RAID is not employed, because both shared partitions must be available for the cluster to run.
c)Do not put the shared partitions on a disk that contains heavily-accessed service data. If possible, locate the shared partitions on disks that contain service data that is rarely accessed.

DD) Make shared partitions and attach it with the cluster :
i) initialise quorum disk once in any node
#mkqdisk -c /dev/sdX -l myqdisk
ii)Add quorum disk to cluster at the backend(In web interface it can be done. Just login into luci interface and go to cluster. You'll see "Quorum Partition" tab. click on it and proceed further to configure it.) :
a)
-----


. . . . .
. . . . .


#expected votes =(nodes total votes + quorum disk votes)



#Health check result is written to quorum disk every 2 secs

#if health check fails over 5 tko, 10 (2*5) secs, the node is rebooted by quorum daemon

#Each heuristic check is run very 2 secs and earn 1 score,if shell script return is 0








-----
Note : Need to manually copy this file on each node. But if you do in web interface, you don't need to manually cop. It'll automatically done.

b)Please increase the config_version by 1 and run ccs_tool update /etc/cluster/cluster.conf.
c) Check to verify that the quorum disk has been initialized correctly : #mkqdisk -L and clustat to check its availability.
d)Please note Total votes=quorum votes=5=2+3, if quorum disk vote is less than (node votes+1), the cluster wouldn’t have survived
e) Typically, the heuristics should be snippets of shell code or commands which help determine a node’s usefulness to the cluster or clients. Ideally, you want to add traces for all of your network paths (e.g. check links, or ping routers), and methods to detect availability of shared storage. Only one master is present at any one time in the cluster, regardless of how many partitions exist within the cluster itself. The master is elected by a simple voting scheme in which the lowest node which believes it is capable of running (i.e. scores high enough) bids for master status. If the other nodes agree, it becomes the master. This
algorithm is run whenever no master is present. Here it is "ping -c1 -t1 10.65.211.86". IP may be san ip/ other nodes' IP etc.


7. Configuring Cluster Daemons :
The Red Hat Cluster Manager provides the following daemons to monitor cluster operation:
cluquorumd — Quorum daemon
clusvcmgrd — Service manager daemon
clurmtabd — Synchronizes NFS mount entries in /var/lib/nfs/rmtab with a private copy on a service's mount point
clulockd — Global lock manager (the only client of this daemon is clusvcmgrd)
clumembd — Membership daemon

8. Configuring Storage : (Either SAN/NAS - using multipath or nfs)
In luci interface click on "add a system" and then go to storage tab and assign the storage in the cluster.


To start the cluster software on a member, type the following commands in this order:

1. service ccsd start
2. service lock_gulmd start or service cman start according to the type of lock manager used
3. service fenced start
4. service clvmd start
5. service gfs start, if you are using Red Hat GFS
6. service rgmanager start

To stop the cluster software on a member, type the following commands in this order:

1. service rgmanager stop
2. service gfs stop, if you are using Red Hat GFS
3. service clvmd stop
4. service fenced stop
5. service lock_gulmd stop or service cman stop according to the type of lock manager used
6. service ccsd stop

Stopping the cluster services on a member causes its services to fail over to an active member.
=================

Testing failover domain (Making availability):

Pre-configuration : Installed httpd on node 68 and 86.
Common home directory : /var/www/html

Configure httpd as failover domain in cluster (in luci): add failover domain > Add resources > Add service and allocate fail over domain and resource to this service.

1. First httpd_service was on 86(allotted resource is ip 67 to httpd(daemon:domain on cluster) )

ip :

[root@vm86 ~]# ip add list|grep inet
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
inet 10.65.211.86/22 brd 10.65.211.255 scope global eth0
inet 10.65.211.67/22 scope global secondary eth0
inet6 fe80::216:3eff:fe74:8d56/64 scope link
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
inet6 fe80::200:ff:fe00:0/64 scope link
[root@vm86 ~]#

2. crashed 86 server ie made down it.

3. httpd service was up : relocated on 68 : Able to access page : http://10.65.211.67:/

IP floated to 68 server : proof

[root@vm68 ~]# ip add list | grep inet
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
inet 10.65.211.68/22 brd 10.65.211.255 scope global eth0
inet 10.65.211.67/22 scope global secondary eth0
inet6 fe80::216:3eff:fe74:8d44/64 scope link
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
inet6 fe80::20
================

Read More

"cluster is not quorate. refusing connection"

Guys,

Environment : Red Hat Enterprise Linux 5.6, RHCS
Error : subject line
Issue : I am not sure while I got this error in the system log since quorate is enabled and working fine on non-firewalled machine where SELinux is also disabled. For the two node cluster, all the cluster.conf are same. One node was connected to the cluster and other didn't.

Resolution :

1. Make sure : chkcofing cman off; chkcofing clvmd off; chkcofing rgmanager off;
2. Make sure all cluster.conf fules are same.
3. Check if iptables are temporary off
4. Start cman, clvmd and rgmanager manually one by one

or

I rebooted whole node and it worked like charm :)

Read More

How add FTP user from backend in linux?

Use following commands :

Environment : RHEL 6, vsftpd

[root@vm91 ~]# useradd -m testing -G users,ftp,wheel -s /bin/bash
[root@vm91 ~]# passwd testing
Changing password for user testing.
New password:
BAD PASSWORD: it is based on a dictionary word
BAD PASSWORD: is too simple
Retype new password:
passwd: all authentication tokens updated successfully.
[root@vm91 ~]#

[root@vm91 ~]# ll -dr /home/*|grep testing
drwx------. 2 testing testing 4096 May 11 06:58 /home/testing
[root@vm91 ~]#

Test the settings :

You may get following error :

[kmaiti@kmaiti ~]$ ftp IP_FTP_server
Connected to FTP_server (*****).
220 (vsFTPd 2.2.2)
Name (*****:kmaiti): testing
331 Please specify the password.
Password:
500 OOPS: cannot change directory:/home/testing

---

just do like : # getsebool ftp_home_dir
then #setsebool -P ftp_home_dir on

Then retry to access the FTP server :

-----
[kmaiti@kmaiti ~]$ ftp FTP_IP
Connected to ser (ser).
220 (vsFTPd 2.2.2)
Name (ser:kmaiti): testing
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>
-----

Try :)

Read More

How to make persistant static route

Environment : All RHEL

Steps :

1. vi /etc/sysconfig/network-scripts/route-ethX and add following :

---
GATEWAY=xxx.xxx.xxx.xxx
NETMASK=yyy.yyy.yyy.yyy
ADDRESS=zzz.zzz.zzz.zzz
---

NB: Replace the address here.

2. service network restart
3. If you use bond0 device : add following entries in /etc/sysconfig/network-scripts/route-

---
default via X.X.X.X dev bond0
10.10.10.0/24 via X.X.X.X dev bond0
---

NB: X.X.X.X is the gateway IP address

try :)

Read More

How to create network bonding device?

Environment : RHEL 6

Steps :

1. vi /etc/modprobe.conf and add : alias bond bonding
2. vi /etc/sysconfig/network-scripts/ifcfg-bond and add :

---
DEVICE=bond0
IPADDR=192.168.1.1
NETMASK=255.255.255.0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
BONDING_OPTS=""
---

NB : N -> 0,1, ...

2. Let two ethernet cards are there like : eth0 and eth1 :They will look like :

----
DEVICE=eth<0/1>
BOOTPROTO=none
ONBOOT=yes
MASTER=bond
SLAVE=yes
USERCTL=no
----

3. Make sure "bonding" kernel module is present on server: lsmod | grep bonding; modprobe bonding;
4. Restart the service network and make the bond0 up.like : #service network restart

Try :)

Read More

How to analysis coredump using GDB tool?

Hello,

Usually particular program or kernel creates core dump/vmcore file. You need to enable coredump on the linux machine. To collect the vmcore file(memory dump using crash kernel) you need to install kexec/kdump utils in the machine.

Once setup is done, you can install GDB like : #yum install gdb

1. Analysis coredump generated by process/binary :

a)Install the particular package which contains that binary. Check the version to make it similar if you want to analysis the coredump file on another system.
b) Install -debuginfo- on the machine of that same package.
c) start GDBing like : #gdb
d) type #bt at the gdb prompt to get the stack of the symbols. Now analyse these to get the clue.

2. To analysis the vmcore file you need to replace the with and with file

then type bt to get backstrace.

Example to gdb on a simple c program :

-----
1. Program :
hello.c

#include

char hello[] = { "Hello, World!" };

int
main()
{
fprintf (stdout, "%s\n", hello);
return (0);
}
-----

2. Compile the above program :

#gcc -g -o hello hello.c

3. Run gdb on the hello binary, i.e. gdb hello.
#gdb hello

4. Some things can be done even before execution is started. The variable hello is global, so it can be seen even before the main procedure starts:

gdb) p hello
$1 = "Hello, World!"
(gdb) p hello[0]
$2 = 72 'H'
(gdb) p *hello
$3 = 72 'H'
(gdb)

5.Next, list the source:

(gdb) l OR gdb list

1 #include
2
3 char hello[] = { "Hello, World!" };
4
5 int
6 main()
7 {
8 fprintf (stdout, "%s\n", hello);
9 return (0);
10 }

6. The list reveals that the fprintf call is on line 8. Apply a breakpoint on that line and resume the code:

(gdb) br 8
Breakpoint 1 at 0x80483ed: file hello.c, line 8.
(gdb) r
Starting program: /home/moller/tinkering/gdb-manual/hello

Breakpoint 1, main () at hello.c:8
8 fprintf (stdout, "%s\n", hello);

7. Finally, use the “next” command to step past the fprintf call, executing it:
gdb n
=======

Try :)

Read More

What is Proxy and Reverse Proxy?

Web Proxies :

A proxy server is a gateway for users to the Web at large. Users configure the proxy in their browser settings, and all HTTP requests are routed via the proxy. Proxies are typically operated by ISPs and network administrators, and serve several purposes: for example,

* to speed access to the Web by caching pages fetched, so that popular pages don't have to be re-fetched for every user who views them.
* to enable controlled access to the web for users behind a firewall.
* to filter or transform web content.

Reverse Proxies :

A reverse proxy is a gateway for servers, and enables one web server to provide content from another transparently. As with a standard proxy, a reverse proxy may serve to improve performance of the web by caching; this is a simple way to mirror a website. Loadbalancing a heavy-duty application, or protecting a vulnerable one, are other common usages. But the most common reason to run a reverse proxy is to enable controlled access from the Web at large to servers behind a firewall.

The proxied server may be a webserver itself, or it may be an application server using a different protocol, or an application server with just rudimentary HTTP that needs to be shielded from the web at large. Since 2004, reverse proxying has been the preferred method of deploying JAVA/Tomcat applications on the Web, replacing the old mod_jk (itself a special-purpose reverse proxy module).

A Reverse Proxy Scenario:

Company example.com has a website at www.example.com, which has a public IP address and DNS entry, and can be accessed from anywhere on the Internet.

The company also has a couple of application servers which have private IP addresses and unregistered DNS entries, and are inside the firewall. The application servers are visible within the network - including the webserver, as "internal1.example.com" and "internal2.example.com", But because they have no public DNS entries, anyone looking at internal1.example.com from outside the company network will get a "no such host" error.

A decision is taken to enable Web access to the application servers. But they should not be exposed to the Internet directly, instead they should be integrated with the webserver, so that http://www.example.com/app1/any-path-here is mapped internally to http://internal1.example.com/any-path-here and http://www.example.com/app2/other-path-here is mapped internally to http://internal2.example.com/other-path-here. This is a typical reverse-proxy situation.

Load following Apache Proxy Modules :

* mod_proxy: The core module deals with proxy infrastructure and configuration and managing a proxy request.
* mod_proxy_http: This handles fetching documents with HTTP and HTTPS.
* mod_proxy_ftp: This handles fetching documents with FTP.
* mod_proxy_connect: This handles the CONNECT method for secure (SSL) tunnelling.
* mod_proxy_ajp: This handles the AJP protocol for Tomcat and similar backend servers.
* mod_proxy_balancer implements clustering and load-balancing over multiple backends.
* mod_cache, mod_disk_cache, mod_mem_cache: These deal with managing a document cache. To enable caching requires mod_cache and one or both of disk_cache and mem_cache.
* mod_proxy_html: This rewrites HTML links into a proxy's address space.
* mod_xml2enc: This supports internationalisation (i18n) on behalf of mod_proxy_html and other markup-filtering modules. space.
* mod_headers: This modifies HTTP request and response headers.
* mod_deflate: Negotiates compression with clients and backends.

Most important are mod_proxy_balancer, mod_cache, mod_disk_cache, mod_mem_cache, mod_deflate

Building Apache for Proxying :
Use options during compiling apache using source code :

----
$ ./configure --enable-so --enable-mods-shared="proxy cache ssl all"
----

Using apxs tool on existing apache installation :

---
apxs -c -i [module-name].c
---

Configuring the Proxy :

Load following modules in http.conf :

----
oadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
#LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
#LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule headers_module modules/mod_headers.so
LoadModule deflate_module modules/mod_deflate.so
LoadFile /usr/lib/libxml2.so
LoadModule xml2enc_module modules/mod_xml2enc.so
LoadModule proxy_html_module modules/mod_proxy_html.so
----

The fundamental configuration directive to set up a reverse proxy is ProxyPass. We use it to set up proxy rules for each of the application servers in the httpd.conf file on the webserver:

----
ProxyPass /app1/ http://internal1.example.com/
ProxyPass /app2/ http://internal2.example.com/
----

However, this is not the whole story. ProxyPass just sends traffic straight through. So when the application servers generate references to themselves (or to other internal addresses), they will be passed straight through to the outside world, where they won't work. The proxy needs to re-map the Location header to its own address space and return a valid URL. The command to enable such rewrites in the HTTP Headers is ProxyPassReverse. The Apache documentation suggests the form:

----
ProxyPassReverse /app1/ http://internal1.example.com/
ProxyPassReverse /app2/ http://internal2.example.com/
----

Note : just made it more available in net. ref : http://www.blogger.com/post-create.g?blogID=4333613427966952010

Read More