HY-533 is a seminar course in the area of ​​Computer Networks, which will focus on Security issues, Privacy, and Intelligence on the Internet. It will cover security issues faced by Internet Service Providers , ie large networks such as those of OTE and FORTHNET in Greece, of Deutsche Telekom in Europe, and AT & T worldwide. Also, the course will focus on modern research (and industrial) trends to address such problems by using Software Defined Network (SDN) technologies that make computer networks "Smarter". Additionally, the course will cover research trends related to data Transparency of Internet users (this concerns the data transparency both of the end-user as well as of entire companies / networks). 

Another aim of the course is to introduce students to modern techniques for the cultivation of innovative thinking and research, with the ultimate objective of linking research, innovation and entrepreneurship. For this purpose, design thinking methodology will be used. Students will apply the methodology through working in teams in a challenge that will be relative to the course’s literature. A mentor will be assigned to every team in order to advise students.

Finally, the course’s objective is that students are able to work together in teams and communicate effectively , both in writing and orally. 

For this purpose, there will be 3-4 lectures on the topics: 

• How to effectively present research?

• How to write a scientific article

• How to work effectively in a team?

CS-543 Software Systems and Technologies for Big Data Applications.

Goals

• Basic principles of modern big data processing frameworks.
• Programming and use of such frameworks depending on the desired functionality: storage, querying, batch processing, graph processing, streaming, deep learning.
• Performance optimizations from the use of those frameworks.
  

Prerequisites

ΗΥ360 και ΗΥ252, or instructor permission.

Instructor

Christos Kozanitis
kozanitis [papaki] ics.forth.gr

Teaching Assistant

Aggelos Sinogeorgos

csdp1266 [papaki] csd.uoc.gr

Class meetings

Monday - Wednesday 16:00 - 18:00.

Classroom

H.206

Office hours

• Instructor: schedule via email. Please include text "543" in the subject of your email.
• TA: schedule via email

Course Text

Assigned paper readings
Online documentation of technologies of interest

Grading

• Class participation - reading discussion (30%)
• Programming assignments (30%)
• Project (40%)
• min grade requirements apply:
  • A homework average of 50%
  • 50% at every project deliverable (proposal, oral presentation, final report)

Computation platform

Cloud credits by AWS. Students will have a credit to use compute and storage services of the Amazon cloud. Registered students should use the submission folder of the first week of the class to send their uoc email address to receive access to the platform.

Course Material

Introduction

• Big Data and Data Science
• A Guide to functional programming with Scala

Apache Spark Architecture and programming model

• Spark architecture
• RDDs
• Spark operators
• lazy evaluation
• Spark SQL
• Spark tutorial + debugging advice

Introduction to Machine Learning

• Brief introduction
• terminology
• supervised vs unsupervised learning
• example pipelines
• linear algebra review

Distributed Machine Learning

• Scalability challenges for common problems: linear regression, logistic regression
• Sparsity
• Spark MLlib
• Non numeric features: One Hot Encoding (OHE)
• OHE Sparsity
• Dimensionality reduction
• Multi dimensional data

Graph Processing

• Challenges of graph processing
• Giraph
• Graphlab
• Graphx

Streaming

• Streaming use cases
• Storm
• Spark Streaming  - Structured Streaming

Storage Systems

• HDFS
• NoSQL
• Cassandra
• Hbase
• MongoDB

Data Representation

• Serialization, Deserialization
• Avro, Thrift, Protocol Buffers
• Column storage
• Dremel
• Parquet

Cluster Management

• Mesos
• Yarn

Deep Learning

• Introduction
• Scale up vs scale out
• MNIST image recognition
• Tensor Flow

Big Data requires the storage, organization, and processing of data at a scale and efficiency -typically of heterogeneous nature and in streaming flow- that go well beyond the capabilities of conventional information technologies. Such requirements have been first introduced for processing the web, and they are today a common place in many industries. In this respect many traditional assumptions break, new query and programming interfaces are required (Map/Reduce), and new computing models will emerge (Cloud Computing). This course aims to introduce parallel/distributed data processing using the MapReduce (M/R) paradigm and provide insights for developing applications on top of the Hadoop platform.

Big data raises also new challenges in data mining. Given the scale and speed of data that needs to be processed as well the variety of parameters to be taken into account, state of the art machine learning algorithms working offline and expecting homogeneous and clean data are also challenged. There is on ongoing effort to design Big Data Mining algorithms accommodating a parallel/distributed or even a streaming evaluation. Of course such kind of incremental, partial evaluation impacts the quality of obtained statistical models and thus algorithms compromise between quality of the learning and computation time. The course will adopt an algorithmic viewpoint: data mining is about applying algorithms to data, rather than using data to “train” a machine-learning engine of some sort.

The course will consist of lectures based both on textbook material (freely-available for download on the Web) and scientific papers. It will also include programming assignments that will provide students with hands-on experience on building data-intensive applications using existing Big Data tools and platforms. The intended audience of this course is MSc and PhD students but also practitioners who plan to design or develop state-of-the-art algorithms available today for Big Data analysis.