Basics

Wednesday, August 13, 2014

inode deleting table entry for item pointing to free inode/ or used inode

Force XFS to repair!

$xfs_repair -L /dev/mapper/Data   

And got following inode error about pointing to either used block or free block

Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
data fork in regular inode 4497195 claims used block 4832288037
data fork in regular inode 349982619 claims used block 4832288030
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- agno = 16
- agno = 17
- agno = 18
- agno = 19
- agno = 20
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 3
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 6
- agno = 14
- agno = 4
- agno = 5
- agno = 16
- agno = 2
- agno = 19
- agno = 20
- agno = 13
- agno = 17
- agno = 18
- agno = 15
data fork in regular inode 4497195 claims used block 4832288037
data fork in regular inode 349982619 claims used block 4832288030
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done




Very hesitantly, I ran xfs_db on the two inodes that xfs_repair complained about. Then ran xfs_repair again, and this time it didn't mention them.... and now I can delete the files I could not delete before...

Hopefully nothing else is missing...

(in my case below instead of mapper/data its md0)

$ xfs_db -x -c 'inode 4497195' -c 'write core.nextents 0' -c 'write core.size 0' /dev/mapper/Data

core.nextents = 0
core.size = 0


$ xfs_db -x -c 'inode 349982619' -c 'write core.nextents 0' -c 'write core.size 0' /dev/mapper/Data

core.nextents = 0
core.size = 0

Thursday, July 24, 2014

Install and setup zabbix agent agentd on RHEL RHEL6.5 UBUNTU UBUNTU12.04 / 14.04

Installing the Zabbix (Monitoring) agent.

RHEL 6.5

<pre>

# Download and install repository package, install keys and add to repolist.

rpm -ivh http://repo.zabbix.com/zabbix/2.2/rhel/6/x86_64/zabbix-release-2.2-1.el6.noarch.rpm

yum install -y zabbix-agent-2.2.4-1.el6.x86_64

# Edit file *.conf file and add following details

sed -i -e "s/Hostname=$.*$/Hostname=`hostname|awk -F. '{print $1}'`/g"

sed -i -e "s/Server=$.*$/Server=10.0.0.22/g" /etc/zabbix/zabbix_agentd.conf

sed -i -e "s/ServerActive=$.*$/ServerActive=10.0.0.22/g" /etc/zabbix/zabbix_agentd.conf

# View the changes that were made to file

grep Hostname= /etc/zabbix/zabbix_agentd.conf

grep Server= /etc/zabbix/zabbix_agentd.conf

grep ServerActive= /etc/zabbix/zabbix_agentd.conf

chkconfig --levels 345 zabbix-agent on

service zabbix-agent restart

</pre>

UBUNTU 12.04

<pre>

# Download and install repository package, install keys and add to repolist.

wget http://repo.zabbix.com/zabbix/2.2/ubuntu/pool/main/z/zabbix-release/zabbix-release_2.2-1+precise_all.deb

dpkg -i zabbix-release_2.2-1+precise_all.deb

# As 'root' Comment out the extras sources from /etc/apt/sources.list file

apt-get update

apt-get install zabbix-agent

# Update the rc configs to set the runlevel for zabbix-agent

update-rc.d zabbix-agent start 20 2 3 4 5 . stop 20 0 1 6 .

# Edit file *.conf file and add following details

sed -i -e "s/Hostname=$.*$/Hostname=`hostname|awk -F. '{print $1}'`/g"

sed -i -e "s/Server=$.*$/Server=10.0.0.22/g" /etc/zabbix/zabbix_agentd.conf

sed -i -e "s/ServerActive=$.*$/ServerActive=10.0.0.22/g" /etc/zabbix/zabbix_agentd.conf

# View the changes that were made to file

grep Hostname= /etc/zabbix/zabbix_agentd.conf

grep Server= /etc/zabbix/zabbix_agentd.conf

grep ServerActive= /etc/zabbix/zabbix_agentd.conf

</pre>

UBUNTU 14.04

<pre>

# Download and install repository package, install keys and add to repolist.

wget http://repo.zabbix.com/zabbix/2.2/ubuntu/pool/main/z/zabbix-release/zabbix-release_2.2-1+trusty_all.deb

dpkg -i zabbix-release_2.2-1+trusty_all.deb

apt-get update

apt-get install zabbix-agent

# Update the rc configs to set the runlevel for zabbix-agent

update-rc.d zabbix-agent start 20 2 3 4 5 . stop 20 0 1 6 .

# Edit file *.conf file and add following details

sed -i -e "s/Hostname=$.*$/Hostname=`hostname|awk -F. '{print $1}'`/g"

sed -i -e "s/Server=$.*$/Server=10.0.0.22/g" /etc/zabbix/zabbix_agentd.conf

sed -i -e "s/ServerActive=$.*$/ServerActive=10.0.0.22/g" /etc/zabbix/zabbix_agentd.conf

# View the changes that were made to file

grep Hostname= /etc/zabbix/zabbix_agentd.conf

grep Server= /etc/zabbix/zabbix_agentd.conf

grep ServerActive= /etc/zabbix/zabbix_agentd.conf

</pre>

WINDOWS 7/8

è Start -> RUN -> type following "command"

***** open commandline as a Admin user

è Add all the new boxes to zabbix

o Download

<pre>

net use N: \\cuda-fs0\installers
mkdir c:\zabbix
xcopy n:\zabbix c:\zabbix /E /Y

</pre>

o Setup the zabbix deamon as service

<pre>

zabbix_agentd.exe --config c:\zabbix\conf\zabbix_agentd.win.conf --install

</pre>

o Adds firewall exception for

§ inbound rules

<pre>

netsh advfirewall firewall add rule localport=10050 protocol=tcp name="Allow zabbix-agent" action=allow dir=in security=notrequired

</pre>

§ outbound rules

<pre> netsh advfirewall firewall add rule localport=10050 protocol=tcp name="Allow zabbix-agent" action=allow dir=out security=notrequired

</pre>

o Start Daemon

net start "zabbix agent"

è Append the configuration file "C:\zabbix\conf\zabbix-agentd.conf"

o Find and replace with following lines (Server/ServerActive=10.0.0.21) without quotes

§ Hostname=eris-w764-buildxx

§ Server=10.0.0.21

§ ServerActive=10.0.0.21

<pre>

hostname.exe > __hostname.txt
set /p host=<__hostname.txt
echo Hostname=%host% >> c:\zabbix\conf\zabbix_agentd.win.conf
echo Server=10.31.97.194 >> c:\zabbix\conf\zabbix_agentd.win.conf
echo ServerActive=10.31.97.194 >> c:\zabbix\conf\zabbix_agentd.win.conf

</pre>

net use N: /DELETE

Friday, July 18, 2014

Turn on SSL in Tomcat7 / Setup SSL on TOMCAT7, Linux, UBUNTU server

The SSL was successfully setup on the APPSERVER.COMPANY.COM server today. I have done a little write up on the steps to setup for a tomcat7 – linux(Ubuntu) server as follows and for future reference.

NOTE: follow examples of commands below closely for the consistency of –alias used in the keystore.

1. Generate Key in keystore with JKS format (default).

Make sure the field "First and Last name" contains the servername or servers fully qualified domain name (FQDN)

Duplicate other fields as it appears below (certs are case sensitive)

root@appserver:/etc/tomcat7# keytool -genkey -keyalg RSA -alias tomcat-fqdn -keystore keystore.jks -validity 999 -keysize 2048

Enter keystore password:

Re-enter new password:

What is your first and last name?

[Unknown]: appserver.company.com

What is the name of your organizational unit?

[Unknown]: SW-CUDA

What is the name of your organization?

[Unknown]: Company Corporation

What is the name of your City or Locality?

[Unknown]: Santa Cruz

What is the name of your State or Province?

[Unknown]: California

What is the two-letter country code for this unit?

[Unknown]: US

Is CN=appserver.company.com, OU=SW, O=Company Corporation, L=Santa Cruz, ST=California, C=US correct?

[no]: yes

Enter key password for <tomcat-fqdn>

(RETURN if same as keystore password):

2. Check the keystore for the list of keys you added, setup alias for listing keys it comes in handy notice "no filename" in alias.

root@appserver:/etc/tomcat7# alias listkey='keytool -list -storepass urpass -keystore '

root@appserver:/etc/tomcat7# listkey keystore.jks

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 1 entry

tomcat-fqdn, Jul 18, 2014, PrivateKeyEntry,

Certificate fingerprint (SHA1): 51:F8:A8:31:BC:63:5A:BD:8E:A5:3B:29:0D:4B:98:03:8C:2A:63:C5

root@appserver:/etc/tomcat7#

3. Configure the tomcat7 configuration file "server.xml" in "/etc/tomcat7/" directory to the keystore.jks path

...

<Connector

protocol="HTTP/1.1"

port="443"

maxThreads="150"

scheme="https"

secure="true"

SSLEnabled="true"

keystoreFile="/etc/tomcat7/keystore.jks"

keystorePass="urpass"

clientAuth="false"

keyAlias="tomcat-fqdn"

sslProtocol="TLS"

...

4. Restart the tomcat server, At this point the https will start working.

P.S: use the following alias, it comes in handy

alias tomreboot='/etc/init.d/tomcat7 stop ; /etc/init.d/tomcat7 start'

root@appserver:/etc/tomcat7# tomreboot

* Stopping Tomcat servlet engine tomcat7 [ OK ]

* Starting Tomcat servlet engine tomcat7 [ OK ]

root@appserver:/etc/tomcat7#

*. Import the Root Cert

root@appserver:/etc/tomcat7# keytool -import -alias root -keystore keystore.jks -trustcacerts -file /root/certs/root.cer

Enter keystore password:

Owner: CN=HQSERVER-NAME, DC=company, DC=com

Issuer: CN=HQSERVER1-NAMERTCA

Serial number: 61. . . . 0004

Valid from: Tue Jan 24 13:36:39 PST 2012 until: Mon Jan 24 13:46:39 PST 2022

...

Trust this certificate? [no]: yes

Certificate was added to keystore

root@appserver:/etc/tomcat7#

Follow Step 2. as follows

root@appserver:/etc/tomcat7# listkey keystore.jks

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 2 entries

root, Jul 18, 2014, trustedCertEntry,

Certificate fingerprint (SHA1): 37:D2:D6:64:7E:40:51:94:43:63:E8:3B:6C: BD:8E:A5:3B:29:0D:4B

tomcat-fqdn, Jul 18, 2014, PrivateKeyEntry,

Certificate fingerprint (SHA1): 51:F8:A8:31:BC:63:5A: 51:F8:A8:31:BC:63:5A:51:F8:A8:31:BC:63

root@appserver:/etc/tomcat7#

4. Generate Certificate Signing Request (CSR) for the Certificate Athority to approve.

root@appserver:/etc/tomcat7# keytool -certreq -keyalg RSA -keysize 2048 -alias tomcat-fqdn -file appserver.company.com.csr -keystore keystore.jks

Enter keystore password:

P.S: Verify the new file named "appserver.company.com.csr" got created.

root@appserver:/etc/tomcat7# ls

Catalina catalina.properties context.xml appserver.company.com.csr keystore.jks logging.properties policy.d server.xml tomcat-users.xml web.xml

root@appserver:/etc/tomcat7#

5. Send over the ".csr" file to the Signing Athority

First will take the .csr and will return a .crt file in return.

If requested he will provide the .cer file as well.

**NOTE: All the files are in binary format.

6. Convert the files from binary to text with base-64 encoding

No special tool is needed for this task.

Double click on the certificate file provided to you in a Windows 7/8 machine.

Click -> "Details" tab --> "Copy to file" button

-> Next

--> select "Base-64 encoded" radiobutton

-> Next

--> "Browse" button and select a location and "name.cer" for the file

-> Finish

7. Do the above steps for both the certificate files and combine the content in seperate file

a. the "root.cer"

b. the "appserver.company.com.cer"

- Create a new file "combo.cer" and copy paste the content from both the files into the new file in seperate lines

- Copy over the combo.cer over to the appserver.company.com servers "/etc/tomcat7/" directory.

8. copy and IMPORT the "combo.cer" file over to the the linux server wit tomcat server

To import the combo.cer into the "keystore.jks" type following command

root@appserver:/etc/tomcat7# keytool -import -alias tomcat-fqdn -keystore keystore.jks -file combo.cer

Enter keystore password:

Certificate reply was installed in keystore

root@appserver:/etc/tomcat7#

Follow the step 2. List the keys, Notice how the "tomcat-fqdn" key has changed after the succesful import

root@appserver:/etc/tomcat7# listkey keystore.jks

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 2 entries

root, Jul 18, 2014, trustedCertEntry,

Certificate fingerprint (SHA1): 37:D2:D6:64:7E:40:51: 37:D2:D6:64:7E:40:51: 37:D2:D6:64:7E:40

tomcat-fqdn, Jul 18, 2014, PrivateKeyEntry,

Certificate fingerprint (SHA1): 51:F8:A8:31:BC:63:5A: 51:F8:A8:31:BC:63:5A:51:F8:A8:31:BC:63

root@appserver:/etc/tomcat7#

9. Repeat Step 4. Reboot the tomcat7 service

Wednesday, July 16, 2014

[NFS] /proc/net/rpc/nfsd file explained

This file is populated writen by /proc/net/rpc/nfsd file by kernel code linux/fs/nfsd/stats.c and linux/net/sunrpc/stats.c:

cat /proc/net/rpc/nfsd
rc 14 277482605 1519481075
fh 5208 0 0 0 0
io 3197575487 677537481
th 128 1196 48 11 677 54 66 31 288 27 26 118
ra 256 1089064507 0 0 0 0 0 0 0 0 0 1410044
net 471073134 0 471089819 60766
rpc 1796938178 0 0 0 0
proc2 18 0 152 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 1 219996357 556065 24059461 43520164 606473 150260668 28056897 545340 198181 3708 0 412742 104649 177365 10 4940 255357 152 3
0 1809695
proc4 2 0 0
proc4ops 40 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Line by line explaination

rc 14 277482605 1519481075
rc <hits> <misses> <nocache> Statistsics for the reply cache [3]
- hits: client it's retransmiting (a bad thing! o hits is good) [1]
- misses: a operation that requires caching
- nocache: a operation that no requires caching

fh 5208 0 0 0 0
fh (filehandle): <stale> <total-lookups> <anonlookups> [1]
<dir-not-in-cache> <nodir-not-in-cache>
- stale: *supose* to be file handle errors (like when you resize the
underlying filesystem)
- total-lookups, anonlookups, dir-not-in-cache, nodir-not-in-cache: do
not appear (and I always seen it as zeros). So I supose they are unused.

io 3197575487 677537481
io (input/output): <bytes-read> <bytes-written>
- bytes-read: bytes read directly from disk
- bytes-written: bytes written to disk

th 128 1196 48 11 677 54 66 31 288 27 26 118
th (threads): <threads> <fullcnt> <10%-20%> <20%-30%> ... <90%-100%> <100%>
- threads: number of nfsd threads
- fullcnt: number of times that the last 10% of threads (so all threads) are busy.
- 10%-20% (1196), 20%-30% ... 90%-100%: histogram (in the unit of seconds) of the percentage of threads is used
1196 means there is 1196 seconds in which more than 10% threads (and less than 20% threads) are
used.

ra 256 1089064507 0 0 0 0 0 0 0 0 0 1410044
ra (read-ahead): <cache-size> <10%> <20%> ... <100%> <not-found>
- cache-size: always the double of number threads
- 10%, 20% ... 100%: how deep it found what was looking for. I *suppose*
this means how far the cached block is from the original block that was
first requested.
- not-found: not found in the read-ahead cache

net 1797032444 15813112 1781194739 52706
net: <netcnt> <netudpcnt> <nettcpcnt> <nettcpconn>
- netcnt: counts every read
- netudpcnt: counts every UDP packet it receives
- nettcpcnt: counts every time it receives data from a TCP connection
- nettcpconn: count every TCP connection it receives

rpc 1796938178 0 0 0 0
rpc: <rpccnt> <rpcbadfmt+rpcbadauth+rpcbadclnt> <rpcbadfmt> <rpcbadauth> <rpcbadclnt>
- rpccnt: counts all rpc operations
- rpcbadfmt: counts if while processing a RPC it encounters the
following errors: err_bad_dir, err_bad_rpc, err_bad_prog, err_bad_vers,
err_bad_proc, err_bad
- rpcbadauth: bad authentication. It does not count if you try to mount
from a machine that it's not in your exports file
- rpcbadclnt: unused

proc3 22 1 219996357 556065 24059461 43520164 606473 150260668 28056897 545340 198181 3708 0 412742 104649 177365 10 4940 255357 152 3
0 1809695
This should be the nfsv3 statistic, comparing with the output of nfsstat, we should be able to figure out what are those number for

Server nfs v3:
null getattr setattr lookup access readlink
1 0% 219996357 46% 556065 0% 24059461 5% 43520164 9% 606473 0%
read write create mkdir symlink mknod
150260668 31% 28056899 5% 545340 0% 198181 0% 3708 0% 0 0%
remove rmdir rename link readdir readdirplus
412742 0% 104649 0% 177365 0% 10 0% 4940 0% 255357 0%
fsstat fsinfo pathconf commit
152 0% 3 0% 0 0% 1809696 0%

Looks like field 9 and filed 10 is the read and write count. But I still can not tell the difference between
<netcnt> and this nfs read count.

reference:
[1] http://www.opensubscriber.com/message/nfs@lists.sourceforge.net/7833588.html
[2] http://blog.peacon.co.uk/wiki/Monitoring_NFS_Performance
[3] ftp://82.96.64.7/pub/linux/kernel/people/marcelo/linux-2.4/fs/nfsd/stats.c

Friday, June 13, 2014

Compile Vim 7.4 on CENTOS 6 or RHEL 6 box --enable-gui=yes :help csapprox+gui

The following article covers the steps required to build and install VIM 7.4 on CentOS 6/RHEL 6 system. It was necessary to install VIM 7.3+ version on my CentOS 6 system in order to use VIM spf13

remove any installed vim packages

yum remove $(rpm -qa | grep ^vim)

download VIM version 7.4 from here: ftp://ftp.vim.org/pub/vim/unix/

mkdir -p /opt/pkgs

cd /opt/pkgs

wget ftp://ftp.vim.org/pub/vim/unix/vim-7.4.tar.bz2

tar jxvf vim-*.tar.bz2

rm -f vim-*.tar.bz2

cd vim*

set-up build environment

yum install gcc make ncurses-devel

configure and build VIM sources

./configure --disable-selinux \

--enable-multibyte \

--with-features=huge \

--with-modified-by=Ndosh \

--enable-gui=yes



make

make install

re-hash the environment

hash -r

Tuesday, April 8, 2014

One magic line to setup awesomeness for VIM editor in OSX [lots of plugins]

On you MAC OSX box run the following command in a terminal prompt for all the awesomeness.

$ curl http://j.mp/spf13-vim3 -L -o - | sh

Credits: Rajesh Parvathaneni: https://github.com/rajonrails/spf13-vim

A basic .muttrc for use with GMAIL

# A basic .muttrc for use with GMAIL

#Change the following six lines to match your Gmail account details

set imap_user = "username@gmail.com"
set imap_pass = "your-password"
set smtp_url = "smtp://username@smtp.gmail.com:587/"
set smtp_pass = "your-password"
set from = "username@gmail.com"
set realname = "First Last"

#Change the following line to a different editor you prefer
set editor = "vim"

set folder = "imaps://imap.gmail.com:993"
set spoolfile = "+INBOX"
set postponed="+[Gmail]/Drafts"

set header_cache=~/.mutt/cache/headers
set message_cachedir=~/.mutt/cache/bodies
set certificate_file=~/.mutt/certificates

set move = no

set sort = 'threads'
set sort_aux = 'last-date-received'
set imap_check_subscribed

ignore "Authentication-Results:"
ignore "DomainKey-Signature:"
ignore "DKIM-Signature:"
hdr_order Date From To Cc