As featured on the
How to do Data Science @ command line?
Bash may not the best way to handle all kinds of data! But, there often comes a time when you are provided with a pure Bash environment, such as what you get in the common Linux based super computers and you just want an early result or view of the data before you drive into the real programming, using Python, R and SQL, SPSS, and so on.
Lear more!
Learn to use Bash in processing real-world data sets
Darkside of the data-intensive languages
Expertise in the data-intensive languages also comes at the price of spending a lot of time on them. In contrast, bash scripting is simple, easy to learn and perfect for mining textual data! Particularly if you deal with genomics, microarrays, social networks, life sciences, and so on. It can help you to quickly sort, search, match, replace, clean and optimise various aspect of your data, and you wouldn’t need to go through any tough learning curves.
Say hello to big data!
There are several examples of practical data mining that will have a flow of importing specific data resources into flat text-type files. Bash can run different programs (grep, sort, sed, and so on) on those files, clean, optimise and extract preliminary views (cut, csvlook, view, cat, head, etc.) of the data. There is one part of data mining, which involves unstructured data and then transforming it into a structured one (awk, shell). A scripting language like Bash can be very useful for doing the transformation. We strongly believe, learning and using Bash shell scripting should be the first step if you want to say, Hello Big Data!
Rocket start your data journey!
If you haven’t used Bash before, feel free to skip the projects and get to the tutorials part. Read the tutorials and then come back to the projects again. The tutorial section will introduce with bash scripting, regular expressions, AWK, sed, grep and so on. Finally, it gives you a concise beginner friendly guide to the big data landscape including an overview of the critical Big Data tools such as HDFS, MapReduce, YARN, Flume, Hive and more. The book finishes with a near-complete list of references to all the relevant command line and Big data tools.
Learn Bash and take your first step into the Data Sciences!
What do you learn?
Project 3: AU Crime Data
In this project, mining a historical dataset provided by the AFP we will find different stats on crimes per Australian city (awk, sort, and so on)
Project 4: Text mining
In this project, we will use containing plays and poems stats from the Shakespeare-era and find Shakespeare’s most freq words. Learn awk, Bash functions, and so on !
Tutorials: Bash, awk, regex and so on
If you haven’t used Bash before, don’t worry! The tutorial section will introduce with bash scripting, regular expressions, AWK, sed, grep and so on.
Beyond the text files
Finally, it gives you a concise beginner friendly guide to the big data landscape including an overview of the critical Big Data tools such as HDFS, MapReduce, etc.
Learn to Analyze Data in Bash Shell & Linux
Choose any of the following, available in 3 different formats
Supporting Materials
For the “Learn to analyze data in Bash Shell and Linux” project.