BTSync - an alternative to cloud storage

For a while now, I have been concerned about my reliance on cloud services such as dropbox, evernote, google drive and Spider Oak. I have an active account with all of these services and more and more I worry about the fact that I have given so much of my data to these companies. I don't necessarily believe that these corporations are necessarily doing anything particularly evil with my data and I also don't upload anything terribly sensitive to the cloud. But in any case, that's not to say that quite a bit of the data I do put on the cloud I would still prefer to keep out of the public domain. For this reason I have started to look an alternative service, BitTorrent Sync.

For those unfamiliar with this system, it is basically a service developped by the folks that invented torrenting where you use a similar bit torrent protocol to sync data between different machines. The beauty of this system lies on the fact that all this data is stored locally in all machines involved and all data transfer is done in an encrypted manner. Meaning that you are not giving your data to anyone else, it stays withing all the machines sharing this data, and that's it. Not only that, but much like bit torrents, it can download this data in a really efficient way using direct connections to all machines involved. Lastly, one of the major advantages of this service over dropbox and others is the fact that there is no storage limits beyond how much you can store on your hard drive. Now, that's not to say that there isn't a downside of using this service. The downside is the fact that at least two machines need to be online for the data to successfully sync, so people like me that turn off their computers when they are not being used can't keep files updated all the time. For this reason I have devised a solution, which is to use my ubuntu server as a torrent sync server. BT Sync by its nature has no server, all clients just share the data, however if I add my home server to the loop, it will be updated at all times when it is online, serving as my personal "cloud" storage in a way. In this post I will go through the process of getting my server setup.

To start, download the appropriate tar file from the official website, http://www.bittorrent.com/sync/download

 note that you need to download the file that correspond to your system, whether it is 32 or 64 bits. To find out what your machine is, type:

$ uname -a

if you see a i386 in the response you are using a 32 bit install, while anything showing x86_64 is likely a 64 bit install.

Once I downloaded the appripriate file, unpak it into a folder where you will keep your executable and configuration files. Note that this folder needs to separate from the folder you intend to share since apparently failing to do that will cause a lot of synching issues.

To unpack the tar ball type:

$ tar -xvzf bittorrent_sync_i386.tar.gz

of course, substitute the name of the tar ball if you are using a 64 bit system. Once that's done run the executable with

$ ./btsync

This should prompt  a message indicating even the process id for the BitTorrent Sync application in case you need to restart it. Setting up your shares on linux is done via a web GUI via the default port 8888. One issue I came across the first time I set up my BTSync was that since I'm using a headless server and connecting to it via SSH, I needed to use the web GUI and for some reason could not connect to it right away. Instead, I needed to configure my BTSync to listen to the 8888 port from all ips (0.0.0.0) instead of just 127.0.0.1. To accomplish this you need to type

$ ./btsync --webui.listen 0.0.0.0:8888

Now to use the web GUI, use your favorite browser from a machine that does have a graphical interface and type the your server's ip followed by the port number you selected, in our case port 8888. So if my internal ip is 192.168.0.70 (use ifconfig if you need to find this out), then use

http://192.168.0.70:8888

From there, click on the "add folder" icon and select the folder you would like to sync. Just remember to backup this folder before going further. It's unlikely you will screw up, but just for good measure, you should always back up your data before an action like this.

Once you have added the folders you want synced to your list, click on the "3 dots" icon at the end of the line where the folder is displayed and go to "preferences". There you should see a button to "View Key". Here you will have the opportunity to copy a key for sharing this folder as a "read only" folder, or as a read and write folder.

Install following the instructions once again on another machine you want to sync with your server and this time, instead of clicking on the add folder button use the cog button instead, where you should see the option to "Enter a key..." That's where you will enter the hashed number you saw previously. Find the folder you want to sync and voila.

That's it, it should work and will work over several different installs. One last detail, you may need to set up a cron job or some other way of initializing the btsync executable every time you turn your server online.

edit- One detail I forgot to mention, you will need your clocks to be synchronized for BTsync to work. In case you need instructions on how to do that on your linux machine, use the following link http://www.howtogeek.com/tips/how-to-sync-your-linux-server-time-with-network-time-servers-ntp/

update error on Ubuntu Server 12.04

Recently I came across an annoying error on my headless Ubuntu Server. Nothing really had happened as far as I'm concerned in my home LAN, and all of a sudden only this distro could no longer run an OS update. I did upgrade my internet service recently but I'm not sure if that could be related. Anyway, the error I kept getting for days was:

 

"  temporary failure resolving 'us.archive.ubuntu.com'  "

Err http://us.archive.ubuntu.com precise Release.gpg

Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/precise/Release.gpg

Temporary failure resolving 'us.archive.ubuntu.com'

Anyway, after a bit of googling, I came across a helpful comment by fossfreedom from the UK, which can be found here. The solution was simple for me:

1. Test to see if the problem is your DNS setting or an ISP issue. To do this type on the console: ping -n 8.8.8.8
2. If produces replies you can stop it with Ctrl + C. This indicates that it is not an ISP problem so there is no point in calling them.
3. Therefore, for me it was a DNS problem, which I have been having a lot of lately (likely due to my VPN usage). So I checked /etc/resolv.conf to see if I could find my DNS servers there. That's when I knew what my problem was. The file was empty, so I added the google DNS server.
4. This was done through: echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf > /dev/null
5. After that, I just had to run the update via: sudo apt-get update

Using an SSH Client on Ubuntu 13.04

Since now I have an Ubuntu Server setup at home, I feel the need to use a client from my Ubuntu 13.04 VM. In order to do that I followed this great concise guide I found from ubuntulinuxhelp.com (from garymacritchie)

---

The SSH Client

The SSH Client is the piece of software that you have on the computer you are sitting in front of and typing on. It sends messages to the SSH server which is on the remote computer (such as the webserver). We’ll assume that there is already a valid SSH server up and running that you are trying to connect to.

In Ubuntu, there is a SSH client installed as standard and it is called Open SSH.

To use it, you simply open a terminal and proceed any command with “ssh”.

For example, if you would like to log in to your remote server, just use the command…

ssh username@mysite.myserver.org

However, as this is a blog that likes to keep things simple (and annoy a lot of the die-hard Linux geeks), I’m going to suggest a different tool… PuTTY.

PuTTY is a client program for SSH (among other things) and gives a neat little interface for making SSH connections. It is also in my opinion one of the best pieces of software ever written. It has been around for ever and can also be used on Windows without installation (just runs as a .exe). Learn this little guy and you will never look for another SSH client.

Install PuTTY by typing the following into a terminal…

sudo apt-get install putty

You will then find the PuTTY program among your other internet applications.

(When you start it up, you will be able to enter the ip address or name of the remote computer you wish to contact and click connect. You will then be asked for password etc. If you are going to be using SSH regular, you can save that connection as a profile which makes things easier).

Linux distro sampling - Ubuntu GNOME and Kali Linux

I decided to try a few new distros this week after realizing that my current Ubuntu 12.04 install is wrecked. Before picking what to install next on my dual boot machine I decided to try out a couple of things in a VM form. So I downloaded VirtualBox, installed that, changed my BIOS to be able to run a virtual machine and off I went installing Ubuntu Gnome.

Ubuntu GNOME

I had seen a few videos of the distro and read some reviews of Gnome 3 and thought this looks quite nice, looked nicer than KDE and it seemed a bit more practical and fast than Unity. Well after installing it I'm not nearly as impressed as I thought I would be. I don't like a lot of aspect of this version of gnome. I thought I would like the idea of a desktop environment that's simple, clean and minimalist, but I don't. I like the search function of the DE but it's not even as good as Unity at that. Not having minimize and maximize buttons is annoying even if they are not essential. The color scheme is annoying and I can't find a quick way to remediate that without downloading something. I know this is probably not related to my desktop environment, but the software center in this distro has no search function, and that's just abnoxious. I ended up downloading synaptic which I do like but seriously. I'm also not a big fan of the file manager that comes with this distro, Nautilus in my opinion is superior. All in all I think I would go back to the regular Ubuntu as soons as I can unless I do find something better before I make the jump back.

Kali Linux

After trying to get my old copy of Backtrack 5 running on my newer laptop and having a lot of issues to fix I decided to go ahead and download a more recent version of BT which would be BT5R3. Well I discovered in this process that there is a whole new distro that is the next BT called Kali Linux. I did a bit of reading and despite my natural propensity to avoid change I went ahead and downloaded Kali. First thing I noticed was how much smaller the ISO file is compared to BT5. The OS uses KDE and it isn't nearly as sleek looking as BT5 was, but obviously aesthetics of a distro with this kind of purposes is kind of a silly thing to worry about. The bottom line is that I'm extremely impressed with this distro. I don't think I will ever touch BackTrack again after experiencing Kali. It's faster, lighter, does the job with minimal problems. It also took a lot less configuring to get things I needed working to work such as the network adapter. I would strongly recommend anyone using BT to try Kali next chance they have, it's a wonderful change from BT in the sense that it does what you need it to with minimal to no headache. Great move by the developers.

Ubuntu Server 12.04

Since I'm at it and this was something I did recently I figured I'd post it here too... I mentioned in earlier posts that I'm also working with a server distro on my old laptop. I'm using it headless with Putty on my desktop machine. Right now I have it running a samba server which shares any HD attached to my dock, and this week I installed miniDLNA on it to share media over my network to other machines as well as my PS3 (mainly to my PS3 really).

My experience so far with this distro has been very positive. It's so fast even on my old shitty laptop. Working on it through a command line and no GUI has actually been quite an interesting experience. I'm not going to say it's not a bit more challenging to work on it this way (at least for a noob like me), but I feel like I'm learning quite a bit and it just makes the whole experience "cleaner" if you understand what I mean. It's nice having such a minimalist OS running in my server.

Overall (tl:dr)

Ubuntu GNOME 13.04 - I want my unity back, sorry Gnome 3 but it's just not going to work out.
Ubuntu Server 12.04 - Fast and reliable, what else do you need from a server?
Kali Linux 1.0.3 - An upgrade from BT5, I didn't think it was possible. It was love at first ARP spoof.

Media Server install on Headless Ubuntu Server 12.04

As I mentioned in a previous post, I've been playing around with Ubuntu Server 12.04 on an old laptop of mine. One of the purposes of putting it together is to stream media over my home network. As I mentioned earlier, I'm running this server headless using Putty. So here is a set of instructions I found to work very well with my install. Credit goes to folks at UbuntuForums.org and damo12 for putting this post together. Another potentially useful link in case there are still questions after this post can be found HERE.

 ----------------------------

Guide to Installing and Running MiniDLNA on an Ubuntu Server

I have recently been experimenting with running MiniDLNA on a headless Ubuntu Server and have hit a few snags. After a lot of search and a few pointers from this and other forums I have finally managed to get it running fine. In case anyone else has this problem I have written this guide to hopefully take some of the pain out of installing and configuring an amazing program.


The Hardware

My server is a headless HP Microserver running Ubuntu 10.04.3 LTS 64 bit edition accessed through Putty and Webmin (www.webmin.com). I have also tried this using a virtual setup using Virtual Box and that also worked fine.

As a receiver, I am using an LG HX806SH Blu-ray player connected to my LAN by a Homeplug adapter. I have also tried using Windows Media Player and they both work fine. I would like to try this with VLC to test it on a Linux desktop but configuring VLC to receive streaming media from a server is beyond me right now.


The Server Setup

The server has a standard headless setup including Samba installed (a combination of Windows and Linux machines share files stored on it). If you need any help installing and using Webmin, I suggest you have a look at the excellent guide at http://www.kelvinwong.ca/2010/05/22/...-04-lts-lucid/.


Installing and Configuring MiniDLNA

For some reason, when I tried to install MiniDLNA, through Putty I received error messages. From reading around, it turns out that this is a common problem as some repositories are missing.

In the end, these commands installed the package:

Code:

sudo apt-get install --reinstall python-software-properties && sudo dpkg-reconfigure python-software-properties

sudo add-apt-repository ppa:stedy6/stedy-minidna
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install minidlna

I would also recommend installing the Webmin MiniDLNA module from https://sourceforge.net/projects/minidlnawebmin/

If you are accessing the server through Webmin and are not sure how to install the module, save it to a local location (such as your desktop on your computer). Select the "Webmin Configuration" module which is under the "Webmin" tab. Select the "Webmin Modules" and choose to load the module “From uploaded file". Once the module has been installed it will be automatically be configured and it will be available under the "Servers" tab.

Once this is installed, you need to configure the MiniDLNA conf file located at /etc/minidlna.conf.

You can use your favourite text editor or if you are accessing this through Webmin, you can use the "Edit" option in "File Manager". If you have installed the Webmin module, you can also change all of the settings from there.

My minidlna.conf file reads as follows:

 [refer to link for the full text, but the only changes I had to made were adding the following]

# set this to the directory you want scanned.
# * if have multiple directories, you can have multiple media_dir= lines
# * if you want to restrict a media_dir to a specific content type, you
#   can prepend the type, followed by a comma, to the directory:
#   + "A" for audio  (eg. media_dir=A,/home/jmaggard/Music)
#   + "V" for video  (eg. media_dir=V,/home/jmaggard/Videos)
#   + "P" for images (eg. media_dir=P,/home/jmaggard/Pictures)

media_dir=V,/media/server/server/Media/Films
media_dir=V,/media/server/server/Media/TV
media_dir=V,/media/server/server/Media/Commedy
media_dir=V,/media/server/server/Music/Videos
media_dir=A,/media/server/server/Music/Music
media_dir=P,/media/server/server/Stuff/Pictures

# set this if you want to customize the name that shows up on your clients
friendly_name=MiniDLNA Server

[Basically you need to add the path to the appropriate folders you want to share, and where it says "friendly_name" you can change to something you can easily identify in your network.]

As you can see, I've made a few changes to the standard file:

• I have added various sources of videos
• I have changed the name of the server so I can recognise it easily on the network
• I have moved the location of the database to a location that I backup each day and is shared by Samba (this is for my personal benefit only)
• I have added extra filenames to the "AlbulmArt" section as I usually call the front cover of an album "Front"

Despite the line “# default presentation url is http address on port 80” MiniDLNA does not have a webpage that it can be controlled from. From what I can gather, this was going to be a future feature that was not completed for whatever reason. However, the Webmin module takes care of everything that you need.

From the "Command Shell" in Webmin or through an SSH session using Putty, you can control MiniDLNA using the commands:

Code:

sudo /etc/init.d/minidlna stop
sudo /etc/init.d/minidlna start

At first run, MiniDLNA will scan all of the folders (and their sub-folders) it has been pointed at and make all of these files available. The scan is extremely quick (less than 10 minutes to scan about 12,000 files).

The Webmin module has a "Restart" button but for some reason (on my set-ups) it does not work properly. It will stop MiniDLNA but when it tries to restart it, it fails. Trying to manually start it using the command "sudo /etc/init.d/minidlna start" after using this "Restart" does not seem to work and the only way I have found to resolve this is to restart the whole server.


Adding or Removing Files of Folders from the Database

In my experience, MiniDLNA tends not to notice if a file has been added or removed from the folders so the database does not update and new files are not shown on your device and old files are still shown even though they have been removed. This seems to be a problem with MiniDLNA and not the receiving soft/hardware. If you change the name of the server, the soft/hardware picks that up fine.

Some people have had success with the command:

Code:

minidlna -R

or by removing the file /tmp/minidlna/files.db and restarting the server using the command:

Code:

rm -rf /tmp/minidlna

Someone else suggested using the command:

Code:

minidlna -R -f /tmp/minidlna.conf

However, I have found the easiest and most successful method is to use the "Rescan" button in the Webmin interface. This button deletes the MiniDLNA database and rescans from scratch. On my system, this new scan took less than 10 minutes to scan about 12,000 files. After the scan, any new files appear in the database and any deleted files are removed.


Outstanding Problems

The only thing I have not been able to get MiniDLNA to do so far is display my photos properly on the Blu-ray player even though they display fine on Windows Media Player. On the Blu-ray, it shows all of the folders where my pictures are stored and even gives me the option to search by camera but when I try to view the pictures or look into the folders, the folders are all empty. This suggests it is something to do with the Blu-ray player and not MiniDLNA however, to be honest this is not an issue for me as my main aim was to stream videos and music and it works perfectly for that.

If anyone can point me to a walkthrough of how to receive streamed videos and music on VLC I would be grateful so I can test this set-up on that and add it to this guide.

I hope you find this guide useful and it takes away some of the headaches I've had with setting up this excellent program.

After installing Ubuntu Server 12.04 (a noob's perspective)

Hello everyone, I just finished installing Ubuntu Server 12.04.2 LTS on my spare laptop. I basically am looking to put together a home server for fun. I think this post may be useful to people that are absolute noobs to this stuff since I'm an absolute amateur and I think I may bring up some difficulties that other new users may experience as well...

Anyway, I went through the installation of the OS to my laptop. For this I followed fairly simple instructions from "The Official Ubuntu Server Book, 2nd ed." as well as this (http://www.youtube.com/watch?v=0SgX15CpT64) video found on YouTube. There are other videos as well if needed, but honestly the process of installing the OS is fairly straight forward if you have ever installed Ubuntu before. There is one thing I would say though, when you are installing the server to your machine, I recommend you plug your machine to your router directly before you start. I started off trying to use the wifi, and that just wasn't going to happen, I only later realized that my original challenges were due to this simple fact.

Once you finish installing Ubuntu Server on your machine, reboot it and I would recommend running an update. In order to accomplish this you need to type:

sudo apt-get update
sudo aptitude safe-upgrade

This is of course all done after you have logged in with you username and password. By the way, make sure you don't forget any of the passwords you have for your server since I'm pretty sure you can't get them back later. Also if you are really new to Linux, sudo is the command you use to gain root access, which is sort of like having administrator privileges. Anytime you use "sudo" you need to use your password.

I also added vim to my server by typing

sudo apt-get install vim

I think that if you are completely new to linux commands, another important one to learn is

sudo shutdown -h now

which turns off your server, as well the command to restart your machine:

sudo shutdown -r now

The last thing I would like to mention, which is what you will need if you want to run the server headless (ie. without a keyboard or monitor directly connected to the machine), is that you need some kind of software that allows you access to your server. For that, one of the more popular options is the use of putty.exe which can be obtained at http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

Once you download putty.exe, run it. There you will type your Host Name or IP address. The hostname is the machine's name you types during the installation of your OS. Make sure SSH is selected in the connection type. This will obviously only work if you have the SSH stuff installed on your server. Save a name for your session, I'm just using the server's name for reference. At this point you can select your session and click "open".

Click yes to the pop up box that will show up (PuTTY Security Alert). At this point you will get a command prompt box, where you can operate your server as if you were typing directly into it. Quite cool really for anyone like me who has never done this before.