Git commands

git log -p --author="Linus Torvalds"

ZFS Commands

Pool related :

#zpool create [name] [type = mirror, raidz, log, cache, sparse] [disk, file]

#zpool status

#zpool destroy

#zpool scrub

#zpool import | export

#zpool online | offline [device]

#zpool attach | detach [mirror]

#zpool cache

#zpool add pool [type]

#zpool get all

#zpool set [property=value] [pool]

#zpool history

Dataset (Volumn / Filesystem) related :

#zfs create [pool]/[dataset]

#zfs get all [pool]/[dataset]

#zfs set [property=value] [pool]/[dataset]

#zfs list

#zfs snapshot [pool]/[dataset]@[name]

#zfs rollback

#zfs clone

#zfs send | receive

Debugging :

#zdb

Using LLVM based Go compiler - llgo

We are going to use llgo - a compiler for Go, written in Go, and using the LLVM compiler infrastructure. The project is hosted at https://github.com/axw/llgo



1. Install LLVM from source using this guide http://linuxdeveloper.blogspot.in/2012/12/building-llvm-32-from-source.html

2. Download Go 1.0.3 binary packages from http://code.google.com/p/go/downloads/list. The file is named "go1.0.3.linux-amd64.tar.gz" and you have to download it in the "~/Downloads" folder.

3. Extract Go in your home directory

$cd ~
$tar zxvf ~/Downloads/go1.0.3.linux-amd64.tar.gz

4. Add the Go binaries to the path.

$gedit ~/.bashrc

Add the following lines

export GOROOT=$HOME/go
export PATH=$PATH:$GOROOT/bin

Reload the shell environment

$source ~/.bashrc

5. Check if Go is available

$go
Go is a tool for managing Go source code.
Usage:
 go command [arguments]
The commands are:...

Good !

6. Now to build llgo we are going to use "go get" command. These steps are also listed in the llgo's README.md file on their website https://github.com/axw/llgo. Type the following in the shell

$export CGO_CFLAGS="`llvm-config --cflags`"
$export CGO_LDFLAGS="`llvm-config --ldflags` -Wl,-L`llvm-config --libdir` -lLLVM-`llvm-config --version`"
$go get github.com/axw/llgo/llgo

It will automatically build and install the llgo binaries in the "~/go/bin" folder.

7. Check if the "llgo" has been build

$ls ~/go/bin
go  godoc  gofmt  llgo
$llgo
No Go source files were specified

There it is :)

8. Lets test it

$gedit ~/test.go

Add these lines

package main
import "fmt"
func main() {
  fmt.Println("Hello World")
}

Now run on the terminal

$llgo -dump test.go

; ModuleID = 'main'
target datalayout = "e-p:64:64:64..."
target triple = "x86_64-unknown-linux"
%0 = type { i8*, i8* }
....

That's the LLVM assembly code which is then given to LLVM backend to generate the actual code.

Building LLVM 3.2 from source

A short guide on how to compile LLVM 3.2 a low level compiler infrastructure with Clang on Ubuntu 12.04 (64bit).



1. Create a directory for LLVM

$mkdir ~/llvm
$cd ~/llvm

Install required dependencies (Note : my system is already setup for development so I cannot find out the exact packages required, if you know them post them as comments)

$sudo apt-get install build-essential

2. Download the latest version of LLVM sources including clang (C frontend) and compiler RT from http://llvm.org/releases/download.html#3.2

$wget http://llvm.org/releases/3.2/llvm-3.2.src.tar.gz
$wget http://llvm.org/releases/3.2/clang-3.2.src.tar.gz
$wget http://llvm.org/releases/3.2/compiler-rt-3.2.src.tar.gz

3. Extract the downloaded sources

$tar zxvf ./llvm-3.2.src.tar.gz
$tar zxvf ./clang-3.2.src.tar.gz
$tar zxvf ./compiler-rt-3.2.src.tar.gz

4. Move folders to correct location. We start by renaming 'llvm-3.2.src' to 'llvm-3.2' and moving 'clang' inside the LLVM 'tools' folder and compiler-rt under the LLVM 'projects' folder. This is where LLVM expects them to be.

$mv ./llvm-3.2.src ./llvm-3.2
$mv ./clang-3.2.src ./clang
$mv ./clang ./llvm-3.2/tools/
$mv ./compiler-rt-3.2.src ./compiler-rt
$mv ./compiler-rt ./llvm-3.2/projects/

5. Once everything is in place we create a separate folder for the build process

$mkdir ./build
$cd ./build

This is how the folder structure looks like

|-- build (currently we are here)
|-- llvm-3.2
|   |-- projects
|   |   |-- compiler-rt
|   |-- tools
|   |   |-- clang
|-- clang-3.2.src.tar.gz
|-- compiler-rt-3.2.src.tar.gz
|-- llvm-3.2.src.tar.gz

6. Now we start the actual configuration and compilation of LLVM inside the 'build' folder outside of the main source directory so as to keep the main source tree clean

$../llvm-3.2/configure --enable-shared

Note : There are various configuration flags for CPU architecture, optimize builds, threads, etc. Check them with the '--help' option.

$../llvm-3.2/configure --help

If there are any missing packages required to compile LLVM it will ask you here.
7. If everything is good we can go ahead and compile it and also keep track of the time taken. After compilation all binaries are available in the 'build/Release+Asserts/bin' folder and libraries are available in the 'build/Release+Asserts/lib' folder.

$time make -j 3
.....
llvm[0]: ***** Completed Release+Asserts Build
real 31m58.051s
user 52m50.842s
sys 2m37.610s

Great job so far, few more steps to go :)

8. Its good to run some test suites that comes along with LLVM to verify everything is working.

$make check-all

9. To start using LLVM we have to include the LLVM binaries and libraries in our path.
To add the LLVM binaries

$gedit ~/.bashrc

Add this line to the end of the file

export PATH=$PATH:~/llvm/build/Release+Asserts/bin

To add the LLVM libraries

sudo gedit /etc/ld.so.conf.d/llvm.conf

Add this line to the file (replace <your_user_name> with your current username)

/home/<your_user_name>/llvm/build/Release+Asserts/lib

To apply the new settings do

$source ~/.bashrc
$sudo ldconfig

10. To check whether the LLVM binaries and libraries are correctly set

$which clang
~/llvm/build/Release+Asserts/bin/clang
$sudo ldconfig -p | grep LLVM
libLLVM-3.2svn.so (libc6,x86-64) => ~/llvm/build/Release+Asserts/lib/libLLVM-3.2svn.so

11. Test a sample program

$cd ..
$mkdir test
$cd test
$gedit test.c

Add the following lines of a simple C program and save it

#include <stdio.h>
int main(void)
{
 printf("Hello World from LLVM!\n");
 return 0;
}

Compile it using clang the C frontend to LLVM

$clang test.c -o test

Run it

$./test
Hello World from LLVM!

Building Linux kernel the clean way

I like to keep my kernel sources clean and have the kernel build in a separate directory. Here is how I do it.



1. Lets create a "~/kernel" directory for holding everything

$cd ~
$mkdir ~/kernel
$cd ~/kernel

2. Download and extract the kernel source

$wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.6.tar.bz2
$tar jxvf linux-3.6.tar.bz2

3. Create a build directory

$mkdir ~/kernel/build

Now our directory structure looks like:

~/
~/kernel/
~/kernel/build/
~/kernel/linux-3.6.tar.bz2
~/kernel/linux-3.6/

4. We will create a bash function that will automatically create the build folder for us based on the folder name of the kernel source. Add the following to the ~/.bashrc file. Once this is done open a new bash terminal so that the function is available for use.

function kb {
    mkdir -p ~/kernel/build/$(basename $(pwd))
    echo ~/kernel/build/$(basename $(pwd))
}

Running $kb in your shell will create the build folder and return the path of the build folder which we will pass it to "make" in the next step.

5. To build the kernel in the "build" folder follow your normal step except...

$cd linux-3.6
$make distclean
$make defconfig O=`kb`
$make O=`kb`

You can see that a folder called "~/kernel/build/linux-3.6" has been created and all the object files are created in the build folder and our original kernel source folder is kept clean. After the compiling is done the kernel is available in the ~/kernel/build/arch/x86/boot/bzImage or whatever is your architecture folder.

Note :
The keys surrounding the `kb` in the above steps are the backtick keys which are located along with the "~" key on your keyboard.

Extra :
If you download and extract the linux-3.5.tar.bz2 and repeat the above 5th step it will create a ~/kernel/build/linux-3.5 folder and build everything there.

Next time I am going to post how you can fully automate the build with git.

Using ketchup to manage kernel releases

We are going to use ketchup to download Linux kernel releases. Its a great tool written in python to automate a lot of stuff.

1. Install ketchup

$sudo apt-get install ketchup
$mkdir linux
$cd linux

2. Lets check the latest version of Linux kernel for the 3.x series

$ketchup -s 3
3.6

3. Lets download the 3.6 kernel release

$ketchup -r 3.6
.....
gpg: Signature made Monday 01 October 2012 05:25:16 AM IST using RSA key ID 00411886
gpg: Can't check signature: public key not found

ketchup: The GPG key seems not to be in the keyring. Please fix this and try again.
In the case potential malicious kernel code is not a problem,
you can skip the verifying by using --no-gpg.
ketchup: Tarball download failed

4. Oops ! You need to add the GPG key to your keyring for verification since each release is signed (Note that the RSA key ID 00411886 is mentioned in the message itself). Lets do that...

$gpg --recv-keys 00411886
gpg: requesting key 00411886 from hkp server keys.gnupg.net
gpg: key 00411886: public key "Linus Torvalds " imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

6. This looks good. Lets try downloading again.

$ketchup -r 3.6
....
Downloading linux-3.6.tar.sign
--2012-10-07 17:47:06--  http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.6.tar.sign
....
Downloading linux-3.6.tar.xz
--2012-10-07 17:47:06--  http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.6.tar.xz
....

7. It will download, verify, extract the kernel sources and it will also rename the directory to "linux-3.6" to match the kernel version downloaded. Great !

8. Since our original "linux" folder has been renamed to "linux-3.6" we need to back down to parent folder and go back to the renamed "linux-3.6" folder

$cd ..
$cd linux-3.6

9. Now lets try something different. Lets download Linux kernel version 3.5

$ketchup -r 3.5

Note that it only downloads the patch file that it needs to apply/revert to 3.6 that is already downloaded to roll back to 3.5 ! Also it will rename the directory to "linux-3.5". Remember to repeat the above step 8 to go to the renamed "linux-3.5" folder.

Do a git checkout of Dragonfly BSD Hammer2 File System

1. Clone the main Dragonfly BSD git repo (approx. 750MB)

$git clone git://git.dragonflybsd.org/dragonfly.git dragonfly
$cd dragonfly

2. List all branches

$git branch -a
* master
  remotes/origin/DragonFly_RELEASE_1_10
  remotes/origin/DragonFly_RELEASE_1_12
  remotes/origin/DragonFly_RELEASE_1_2
  remotes/origin/DragonFly_RELEASE_1_4
  remotes/origin/DragonFly_RELEASE_1_6
  ....
  remotes/origin/HEAD -> origin/master
  remotes/origin/doc
  remotes/origin/hammer2   <======================== HAMMER2
  remotes/origin/master
  remotes/origin/netmp
  ....

3. Checkout the Hammer2 branch

$git checkout hammer2
cd sys/vfs/hammer2