Breaking - Amazon Web Services

What all comes under Amazon Web Services?

Cloud is becoming more & more critical for most businesses as it provides flexibility, cost-effective on-demand storage. The cloud computing service deployment models give command and control over data.  

Types of service models:

1. Infrastructure as a Service: IaaS lets users use Virtual machines, storage & servers 

2. Platform as a Service: PaaS lets users develop & host apps on the platform 

3. Software as a Service: SaaS lets users access these applications across all devices
4. Network as a Service: NaaS allows users to access network infrastructure directly & securely. Help deploy custom routing protocols, manage & maintain network resources.
5. Identity as a Service: IDaaS manages user's digital identity which can be used during electronic transactions.

AWS has a huge size and presence in the cloud business today. It has two main products 

1. EC2: Virtual machine service 

2. S3: Amazon storage system (Simple Storage System)

Amazon Redshift: Cloud-based Data Warehouse. 

Amazon Glacier: Low-cost cold storage for infrequently accessed data for long retrieval times. 

Amazon Elastic Block Store: Store persistent data in block-level storage which is highly available even when EC2 is shut down.

Amazon Elastic Cloud Compute (EC2): Virtual servers know as instances for running applications. 

Database Management: Relational database service (RDS) for users to migrate and backup data.

Data Migration: To bring data from servers, databases & applications to AWS cloud. Migration Hub & Snowball.

Networking: Secluded segment of AWS cloud and balancing traffic. 

Security: IAM - Identity & Access management to create and control custom policies for multiple accounts.

Amazon Messaging Services: SQS Simple Queue service SNS Simple notification service SES Simple email service. 

Amazon Development tools: Command line interface & SDK's

Breaking - Clustered Computing

What is Clustered Computing in Big data?

To handle big data individual computers are often inadequate when it comes to storage and compute requirements. Node is a single computer and cluster is a bunch of them together.

Clustering helps in combining resources of smaller machines:

1. Resource pooling: Storage, CPU & Memory
2. High availability: Varying levels of fault tolerance to prevent failures
3. Easy scalability: Add machines horizontally

Manual & Automatic cluster examples: Veritas, Linux natic clusters, IBM AIX bases clusters.

One also needs solutions to manage cluster membership, resource sharing & scheduling work on individual nodes. Hadoop's YARN or Apache Mesos.

The cluster acts as a foundation layer for other processing software.  

Breaking Big Data

What is Big data and the key terms under it? 

Big data in simple terms is non-traditional strategies to handle large data sets.

Main characteristics: 

1. Volume- Scale

2. Velocity- Real-time 

3. Variety- Formats (Structure)

4. Veracity- Accuracy (Complexity in processing)

5. Variability- In quality (Additional resources needed to improve data)

6. Value- Reduce complexity to show value

Life cycle:

Ingesting data: Taking raw data and adding to the system

a) Data ingestions tools- Apache Sqoop can take data from relation databases
b) Projects- Apache Flume & Chukwa designed to aggregate & import application & server logs
c) Queuing system- Apache Kafka can act as an interface between data generator & big data system
d) Ingestion frameworks- Apache Goblin can aggregate & normalize the output of the above at the end of the ingestion pipeline

Persisting data: Persistent storage means making data available even when power is off. Like a hard disk. The volume, requirement of availability & distributed compute makes it complex storage. 

Hence the need to leverage a distributed file system like Apache Hadoop's HDFS, which allows large quantities of data to be written across multiple nodes in clusters. Other distributed systems are Ceph & Gluster FS

Computing and analyzing data: This layer depends on the type of insights required. Data is often processed repeatedly, either multiple times by a single tool or by a number of tools to bring different insights.

Batch processing is one of the methods. Basically giving different batches or pieces and giving to different machines. Then using the output calculating and assembling the final result. These steps are referred to as splitting, mapping, shuffling, reducing & assembling or collectively as map-reduce. This is the strategy that Apache Hadoop's MapReduce program for very large data sets. 

Other workloads require more realtime processing. It works with in-memory hence avoiding to write back on disk. Apache Storm, Apache Flink, and Apache Spark provide different ways of achieving real-time processing a strategy suited for small data sets. 

Other ways of computing using tools that plug-in to the above-given frameworks and provide an additional interface. For e.g.

Apache Hive: Datawarehouse for Hadoop 
Apache Pig: High-level querying interface 
Apache Drill, Impala, Spark SQL & Presto: SQL like interactions can be done using these projects
Apache SystemML, Mahout, Spark's MLlib: For Machine learning projects

R & Python: Analytics programming

Visualizing: Recognizing changes over time is more important than the data itself. Visualizing data helps spot trends. Real-time processing is needed to visualize application & server metrics.

Prometheus: Processing data stream as a time-series database and visualizing it.
Elastic stack: Logstash for data collection, Elastic search for indexing data & Kibana for visualization.
Silk: Apache Solr for indexing & Kibana fork called Banana for visualization. 

Another visualization technology used for interactive data science work is a data "notebook". Especially in Formats that are conducive to sharing, presenting & collaborating. Examples of such interface are Jupyter Notebook & Apache Zeppelin.   

Types of TV advertising

What are the different types of TV advertising?

Advanced TV includes all types of Non-traditional formats of TV viewing. 

For e.g. Interactive TV, Connected TV, Smart TV & OTT (Over the top). The most common term is OTT, which comprises of:

Streaming devices - Apple TV, Android Tv, etc.

HDMI sticks - Chromecast, Roku, Fire stick

Smart TVs - TVs with Netflix or HBO Go.

Game consoles

DVR set-top boxes

Smart Blue-ray/DVD players

Advertising on these internet-enabled devices is more targeted.  

Programmatic TV- Advanced TV with Ads purchased programmatically. Either thru streaming provider or Ad-network.

Addressable TV- More targetted ads shown to different households for the same program. Targeting on OTT is mostly done on a household level.

Models used for targeting:

High-index modeling- Ads decided based on viewership data for a particular show.

CrossDevice graph - Data from different devices in a household to create identifiers & target audience. 

a) Probabilistic: Device type, location, time of day. The probability is benchmarked to get expected accuracy. 

b) Deterministic: Login data for the consumer across devices 

c) Look-alike modeling for finding a similar audience to build scale

Revenue models for Video

SVOD - Subscription video on demand. Access to an entire library for a small recurring fee. Netflix, Hulu, HBO.  

TVOD - Transactional video on demand. Pay-per-view basis. Charged per video. Google play or iTunes. 

AVOD - Ad-based-video on demand. Here the content is free. Youtube.

Video Advertising Formats

What are the different types of Video advertising formats?

Video advertising is to show ads either within, before or after the video. They are known as Mid-roll, pre-roll or post-roll ads. All these three types of ads come under Linear ads. 

So they are like normal ads but they play either before the video or in between the video (so the video one is watching is stopped for that duration) or after the video is completed.

Non-Linear Ads run along with the video so the users see the Ad while seeing the video. 

Companion ads: Run on the side and do not disturb the viewing experience. It can be in the form of an image, text or video. 

  

Bid Shading - First price auctions

What is Bid Shading?

A technique in first-party auctions to optimize the bid price without reducing the chance of winning. So basically an algorithm that arrives at a price optimization strategy considering factors like pricing, website, exchange in place as well as a competitive environment.

The shift started happening after Google's announcement of moving to first-party auction model.

Done by DSPs or Exchanges since they moved to first-price auctions. DSPs are in the best position to do Bid-shading as they have a complete picture of the bid data.

Hence taking away any advantage that Publishers had in the short term. 

They look at the holistic view of the competitive environment to arrive at the price for an advertiser while bidding for an impression. So there is a trade-off here, as you loose quality impressions in case you cannot win at the right price. 

Google: Optimize fixed CPM bidding

Rubicon Project: EMR or Estimated market rate 

Pubmatic: Intelligent bidding

Trade desk: KOA

Bid shading has reduced CPMs for publishers and there is more downside in the future as algorithms improve. 

More transparency is required to show the delta in value in cases where Bid shading was applied. 

Types of Targeting in Advertising

What are the different types of Targeting in advertising?

Behavior- If we segment customers basis their behavior on the website like previous purchases, pages visited, searches done, etc. We then group of customers who have similar behavior and then show them targeted messages/ads.

Context- Targeting users based on the page content or site category. So showing ads basis the content on the website as well depending on the category the website belongs to. IAB has a list of categories for websites. Native is a form of contextual advertising.

Time & Event-based- Ads run at certain time and occurrence of a specific event. Gold jewelry ads around Diwali in India.

Demographic- Basis a person Age, Sex, Income, Ethnicity users can be targeted. When behavior segments are layered on top of demographic the targeting can be more valuable.

Geolocation- Targeting basis a person's location or hyper location (geofencing) which is more precise. So if a person is near a certain store h/she can be shown ads or given vouchers for the same retailer.

Device- More relevant to Mobile ads and serving local ads or coupons. This is again similar to contextual targeting in certain ways. Also, affect the type of creative is served. iPhone users with a 5G connection might see very different ads for someone having a 3G connection.

Site Retargeting- Also known as Retargeting, serving ads to users who came to your website and didn't convert. So these users know about your brand hence higher is the probability for them to convert. 

Search Retargeting- Ads served to users basis their search behavior or keyword data. It is intent-based and exchanges help do this kind of targeting.

Predictive Targeting- Using data from different targeting methods predictive targeting comes up with a probability of purchase for users.