Monday, June 23, 2014

Building LLVM 3.4.2 from source

A short guide on how to compile LLVM 3.4 a low level compiler infrastructure with Clang on Ubuntu 12.04 (64bit).

NOTE : This is a update to the guide on building LLVM 3.2 already posted here. Clang is also known as Clang CFE (C Front End).

1. Create a directory for LLVM
$mkdir ~/llvm
$cd ~/llvm
Install required dependencies (Note : my system is already setup for development so I cannot find out the exact packages required, if you know them post them as comments)
$sudo apt-get install build-essential
2. Download the latest version of LLVM sources including clang (C frontend) and compiler RT from http://llvm.org/releases/download.html#3.4.2
$wget http://llvm.org/releases/3.4.2/llvm-3.4.2.src.tar.gz
$wget http://llvm.org/releases/3.4.2/cfe-3.4.2.src.tar.gz
$wget http://llvm.org/releases/3.4/compiler-rt-3.4.src.tar.gz
3. Extract the downloaded sources
$tar zxvf ./llvm-3.4.2.src.tar.gz
$tar zxvf ./cfe-3.4.2.src.tar.gz
$tar zxvf ./compiler-rt-3.4.src.tar.gz
4. Move folders to correct location. We start by renaming 'llvm-3.4.2.src' to 'llvm-3.4.2' and moving 'clang' inside the LLVM 'tools' folder and compiler-rt under the LLVM 'projects' folder. This is where LLVM expects them to be.
$mv ./llvm-3.4.2.src ./llvm-3.4.2
$mv ./cfe-3.4.2.src ./clang
$mv ./clang ./llvm-3.4.2/tools/
$mv ./compiler-rt-3.4 ./compiler-rt
$mv ./compiler-rt ./llvm-3.4.2/projects/
5. Once everything is in place we create a separate folder for the build process
$mkdir ./build
$cd ./build
This is how the folder structure looks like
|-- build (currently we are here)
|-- llvm-3.4.2
|   |-- projects
|   |   |-- compiler-rt
|   |-- tools
|   |   |-- clang
|-- cfe-3.4.2.src.tar.gz
|-- compiler-rt-3.4.src.tar.gz
|-- llvm-3.4.2.src.tar.gz
6. Now we start the actual configuration and compilation of LLVM inside the 'build' folder outside of the main source directory so as to keep the main source tree clean
$../llvm-3.4.2/configure --enable-shared --enable-targets=host-only
Note : There are various configuration flags for CPU architecture, optimize builds, threads, etc. Check them with the '--help' option.
$../llvm-3.4.2/configure --help
If there are any missing packages required to compile LLVM it will ask you here.
7. If everything is good we can go ahead and compile it and also keep track of the time taken. After compilation all binaries are available in the 'build/Release+Asserts/bin' folder and libraries are available in the 'build/Release+Asserts/lib' folder.
$time make -j 3
.....
llvm[0]: ***** Completed Release+Asserts Build
real 64m49.849s
user 61m49.232s
sys 2m46.721s
Great job so far, few more steps to go :)

8. Its good to run some test suites that comes along with LLVM to verify everything is working.
$make check-all
9. To start using LLVM we have to include the LLVM binaries and libraries in our path.
To add the LLVM binaries
$gedit ~/.bashrc
Add this line to the end of the file
export PATH=$PATH:$HOME/llvm/build/Release+Asserts/bin
To add the LLVM libraries
sudo gedit /etc/ld.so.conf.d/llvm.conf
Add this line to the file (replace <your_user_name> with your current username)
/home/<your_user_name>/llvm/build/Release+Asserts/lib
To apply the new settings do
$source ~/.bashrc
$sudo ldconfig
10. To check whether the LLVM binaries and libraries are correctly set
$which clang
~/llvm/build/Release+Asserts/bin/clang
$sudo ldconfig -p | grep LLVM
libLLVM-3.4.so (libc6,x86-64) => /home/prashants/llvm/build/Release+Asserts/lib/libLLVM-3.4.so
11. Test a sample program
$cd ..
$mkdir test
$cd test
$gedit test.c
Add the following lines of a simple C program and save it
#include <stdio.h>
int main(void)
{
 printf("Hello World from LLVM!\n");
 return 0;
}
Compile it using clang the C frontend to LLVM
$clang test.c -o test
Run it
$./test
Hello World from LLVM!

Tuesday, December 10, 2013

Saturday, December 7, 2013

ZFS Commands

Pool related :


#zpool create [name] [type = mirror, raidz, log, cache, sparse] [disk, file]

#zpool status

#zpool destroy

#zpool scrub

#zpool import | export

#zpool online | offline [device]

#zpool attach | detach [mirror]

#zpool cache

#zpool add pool [type]

#zpool get all

#zpool set [property=value] [pool]

#zpool history

Dataset (Volumn / Filesystem) related :


#zfs create [pool]/[dataset]

#zfs get all [pool]/[dataset]

#zfs set [property=value] [pool]/[dataset]

#zfs list

#zfs snapshot [pool]/[dataset]@[name]

#zfs rollback

#zfs clone

#zfs send | receive

Debugging :


#zdb



Thursday, January 3, 2013

Using LLVM based Go compiler - llgo

We are going to use llgo - a compiler for Go, written in Go, and using the LLVM compiler infrastructure. The project is hosted at https://github.com/axw/llgo



1. Install LLVM from source using this guide http://linuxdeveloper.blogspot.in/2012/12/building-llvm-32-from-source.html

2. Download Go 1.0.3 binary packages from http://code.google.com/p/go/downloads/list. The file is named "go1.0.3.linux-amd64.tar.gz" and you have to download it in the "~/Downloads" folder.

3. Extract Go in your home directory
$cd ~
$tar zxvf ~/Downloads/go1.0.3.linux-amd64.tar.gz
4. Add the Go binaries to the path.
$gedit ~/.bashrc
Add the following lines
export GOROOT=$HOME/go
export PATH=$PATH:$GOROOT/bin
Reload the shell environment
$source ~/.bashrc
5. Check if Go is available
$go
Go is a tool for managing Go source code.
Usage:
 go command [arguments]
The commands are:...
Good !

6. Now to build llgo we are going to use "go get" command. These steps are also listed in the llgo's README.md file on their website https://github.com/axw/llgo. Type the following in the shell
$export CGO_CFLAGS="`llvm-config --cflags`"
$export CGO_LDFLAGS="`llvm-config --ldflags` -Wl,-L`llvm-config --libdir` -lLLVM-`llvm-config --version`"
$go get github.com/axw/llgo/llgo
It will automatically build and install the llgo binaries in the "~/go/bin" folder.

7. Check if the "llgo" has been build
$ls ~/go/bin
go  godoc  gofmt  llgo
$llgo
No Go source files were specified
There it is :)

8. Lets test it
$gedit ~/test.go
Add these lines
package main
import "fmt"
func main() {
  fmt.Println("Hello World")
}
Now run on the terminal
$llgo -dump test.go

; ModuleID = 'main'
target datalayout = "e-p:64:64:64..."
target triple = "x86_64-unknown-linux"
%0 = type { i8*, i8* }
....
That's the LLVM assembly code which is then given to LLVM backend to generate the actual code.

Monday, December 31, 2012

Building LLVM 3.2 from source

A short guide on how to compile LLVM 3.2 a low level compiler infrastructure with Clang on Ubuntu 12.04 (64bit).



1. Create a directory for LLVM
$mkdir ~/llvm
$cd ~/llvm
Install required dependencies (Note : my system is already setup for development so I cannot find out the exact packages required, if you know them post them as comments)
$sudo apt-get install build-essential
2. Download the latest version of LLVM sources including clang (C frontend) and compiler RT from http://llvm.org/releases/download.html#3.2
$wget http://llvm.org/releases/3.2/llvm-3.2.src.tar.gz
$wget http://llvm.org/releases/3.2/clang-3.2.src.tar.gz
$wget http://llvm.org/releases/3.2/compiler-rt-3.2.src.tar.gz
3. Extract the downloaded sources
$tar zxvf ./llvm-3.2.src.tar.gz
$tar zxvf ./clang-3.2.src.tar.gz
$tar zxvf ./compiler-rt-3.2.src.tar.gz
4. Move folders to correct location. We start by renaming 'llvm-3.2.src' to 'llvm-3.2' and moving 'clang' inside the LLVM 'tools' folder and compiler-rt under the LLVM 'projects' folder. This is where LLVM expects them to be.
$mv ./llvm-3.2.src ./llvm-3.2
$mv ./clang-3.2.src ./clang
$mv ./clang ./llvm-3.2/tools/
$mv ./compiler-rt-3.2.src ./compiler-rt
$mv ./compiler-rt ./llvm-3.2/projects/
5. Once everything is in place we create a separate folder for the build process
$mkdir ./build
$cd ./build
This is how the folder structure looks like
|-- build (currently we are here)
|-- llvm-3.2
|   |-- projects
|   |   |-- compiler-rt
|   |-- tools
|   |   |-- clang
|-- clang-3.2.src.tar.gz
|-- compiler-rt-3.2.src.tar.gz
|-- llvm-3.2.src.tar.gz
6. Now we start the actual configuration and compilation of LLVM inside the 'build' folder outside of the main source directory so as to keep the main source tree clean
$../llvm-3.2/configure --enable-shared
Note : There are various configuration flags for CPU architecture, optimize builds, threads, etc. Check them with the '--help' option.
$../llvm-3.2/configure --help
If there are any missing packages required to compile LLVM it will ask you here.
7. If everything is good we can go ahead and compile it and also keep track of the time taken. After compilation all binaries are available in the 'build/Release+Asserts/bin' folder and libraries are available in the 'build/Release+Asserts/lib' folder.
$time make -j 3
.....
llvm[0]: ***** Completed Release+Asserts Build
real 31m58.051s
user 52m50.842s
sys 2m37.610s
Great job so far, few more steps to go :)

8. Its good to run some test suites that comes along with LLVM to verify everything is working.
$make check-all
9. To start using LLVM we have to include the LLVM binaries and libraries in our path.
To add the LLVM binaries
$gedit ~/.bashrc
Add this line to the end of the file
export PATH=$PATH:~/llvm/build/Release+Asserts/bin
To add the LLVM libraries
sudo gedit /etc/ld.so.conf.d/llvm.conf
Add this line to the file (replace <your_user_name> with your current username)
/home/<your_user_name>/llvm/build/Release+Asserts/lib
To apply the new settings do
$source ~/.bashrc
$sudo ldconfig
10. To check whether the LLVM binaries and libraries are correctly set
$which clang
~/llvm/build/Release+Asserts/bin/clang
$sudo ldconfig -p | grep LLVM
libLLVM-3.2svn.so (libc6,x86-64) => ~/llvm/build/Release+Asserts/lib/libLLVM-3.2svn.so
11. Test a sample program
$cd ..
$mkdir test
$cd test
$gedit test.c
Add the following lines of a simple C program and save it
#include <stdio.h>
int main(void)
{
 printf("Hello World from LLVM!\n");
 return 0;
}
Compile it using clang the C frontend to LLVM
$clang test.c -o test
Run it
$./test
Hello World from LLVM!

Wednesday, October 10, 2012

Building Linux kernel the clean way

I like to keep my kernel sources clean and have the kernel build in a separate directory. Here is how I do it.



1. Lets create a "~/kernel" directory for holding everything
$cd ~
$mkdir ~/kernel
$cd ~/kernel
2. Download and extract the kernel source
$wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.6.tar.bz2
$tar jxvf linux-3.6.tar.bz2
3. Create a build directory
$mkdir ~/kernel/build
Now our directory structure looks like:
~/
~/kernel/
~/kernel/build/
~/kernel/linux-3.6.tar.bz2
~/kernel/linux-3.6/
4. We will create a bash function that will automatically create the build folder for us based on the folder name of the kernel source. Add the following to the ~/.bashrc file. Once this is done open a new bash terminal so that the function is available for use.
function kb {
    mkdir -p ~/kernel/build/$(basename $(pwd))
    echo ~/kernel/build/$(basename $(pwd))
}
Running $kb in your shell will create the build folder and return the path of the build folder which we will pass it to "make" in the next step.

5. To build the kernel in the "build" folder follow your normal step except...
$cd linux-3.6
$make distclean
$make defconfig O=`kb`
$make O=`kb`
You can see that a folder called "~/kernel/build/linux-3.6" has been created and all the object files are created in the build folder and our original kernel source folder is kept clean. After the compiling is done the kernel is available in the ~/kernel/build/arch/x86/boot/bzImage or whatever is your architecture folder.

Note :
The keys surrounding the `kb` in the above steps are the backtick keys which are located along with the "~" key on your keyboard.

Extra :
If you download and extract the linux-3.5.tar.bz2 and repeat the above 5th step it will create a ~/kernel/build/linux-3.5 folder and build everything there.

Next time I am going to post how you can fully automate the build with git.

Sunday, October 7, 2012

Using ketchup to manage kernel releases

We are going to use ketchup to download Linux kernel releases. Its a great tool written in python to automate a lot of stuff.

1. Install ketchup
$sudo apt-get install ketchup
$mkdir linux
$cd linux
2. Lets check the latest version of Linux kernel for the 3.x series
$ketchup -s 3
3.6
3. Lets download the 3.6 kernel release
$ketchup -r 3.6
.....
gpg: Signature made Monday 01 October 2012 05:25:16 AM IST using RSA key ID 00411886
gpg: Can't check signature: public key not found

ketchup: The GPG key seems not to be in the keyring. Please fix this and try again.
In the case potential malicious kernel code is not a problem,
you can skip the verifying by using --no-gpg.
ketchup: Tarball download failed
4. Oops ! You need to add the GPG key to your keyring for verification since each release is signed (Note that the RSA key ID 00411886 is mentioned in the message itself). Lets do that...
$gpg --recv-keys 00411886
gpg: requesting key 00411886 from hkp server keys.gnupg.net
gpg: key 00411886: public key "Linus Torvalds " imported
gpg: no ultimately trusted keys found
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
6. This looks good. Lets try downloading again.
$ketchup -r 3.6
....
Downloading linux-3.6.tar.sign
--2012-10-07 17:47:06--  http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.6.tar.sign
....
Downloading linux-3.6.tar.xz
--2012-10-07 17:47:06--  http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.6.tar.xz
....
7. It will download, verify, extract the kernel sources and it will also rename the directory to "linux-3.6" to match the kernel version downloaded. Great !

8. Since our original "linux" folder has been renamed to "linux-3.6" we need to back down to parent folder and go back to the renamed "linux-3.6" folder
$cd ..
$cd linux-3.6
9. Now lets try something different. Lets download Linux kernel version 3.5
$ketchup -r 3.5
Note that it only downloads the patch file that it needs to apply/revert to 3.6 that is already downloaded to roll back to 3.5 ! Also it will rename the directory to "linux-3.5". Remember to repeat the above step 8 to go to the renamed "linux-3.5" folder.