Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads.
Taming the Compliance Beast:
Lessons learnt at LinkedIn
Sept 28, 2017
Shirshanka Das, Principal Staff Engineer, LinkedIn
T...
Data Protection in a Digital World
PLAYING CATCH-UP WITH INNOVATION
GDPR
metric scripts
production code
Business facing
decision making
OUR VISION
Create economic opportunity for every
member of ...
The LinkedIn Privacy Paradox
“On one hand, the company has
500+ million members trusting
the company to protect highly
sen...
metric scripts
Members First is a Core Value for LinkedIn
MEMBER PRIVACY WHILE DELIVERING MEMBER VALUE
production code
Wel...
Data Is the Lifeblood of LinkedIn
MEMBER EXPERIENCES + BUSINESS DECISIONS
production code
Member Data
System of Intelligen...
We needed data democracy to
deliver member value
LinkedIn Data Science
I want to analyze as much data as
possible so my mo...
I want my personal data to be stored only
where needed and not propagated
unnecessarily
Data Protection
Need to Ensure Mem...
DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The ...
LinkedIn’s Data Ecosystem
LinkedIn’s Data Ecosystem
LinkedIn’s Data Ecosystem
LinkedIn’s Data Ecosystem
LinkedIn’s Data Ecosystem
LinkedIn’s Data Ecosystem
DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The ...
Data Hubs at LinkedIn
In Motion
At Rest
Scale
O(10) clusters
~2.3 Trillion messages
~450 TB
Scale
O(10) clusters
~10K mach...
In Motion
At Rest
Data Integration
SFTP
JDBC
REST
Azure
Blob, Data
Lake
Storage
SFTP
JDBC
REST
Apache Gobblin: Simplifying Data Integration
@LinkedIn
Hundreds of TB per day
Thousands of datasets
~30 dif...
REQUIREMENTS
Less Data
Legal: Right to Erasure or Right to be Forgotten
“Delete all my personal data without undue delay w...
A lot of data, different formats
Challenges
Understand HDFS data: organization, formats, …
Cycle asynchronously, within an...
Gobblin: The Logical Pipeline
Source
Work
Unit
Work
Unit
Work
Unit
Extract Convert Quality Write Data
Publish
WriteQuality...
Gobblin: Extending for Purge
HDFS
Work
Unit
Data
Publish
Extract Convert Quality Write
Task
Task
HDFS
If needs purge
then ...
STATUS AND CHALLENGES
Gobblin: Data Lifecycle Management at Scale
Status
Number of datasets: many thousands
Amount of data...
DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The ...
DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The ...
LinkedIn’s Data Ecosystem
Metadata based Search Experience
for Data Scientists
Data Discovery
Where is dataset X?
How did it get created?
Usage : In...
SEARCH SCREENSHOTS
WhereHows
LINEAGE SCREENSHOTS
WhereHows
More than just Discovery
Use Cases
Which datasets at LinkedIn contain PII or highly
confidential data?
How many contain me...
Wide + Deep
Metadata
Comprehensive coverage of data systems at LinkedIn
We have > 20 systems!
SQL, NoSQL, Indexes, Blob St...
A METADATA REFINERY APPROACH
WhereHows Architecture @ 10,000 ft
ML driven
refinements
DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The ...
METADATA
DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Ac...
FREEDOM OF EXPRESSION
Many Transformation Engines @ LinkedIn
In Motion
At Rest
HARD TO CHANGE ANYTHING UNDERNEATH!
Challenge for Infrastructure Providers
(Pig scripts)
My Raw Data
Native readers, depen...
HARD TO CHANGE ANYTHING UPSTREAM!
Semantic Challenges
Data is unclean (bad data on certain dates)
Data models are in const...
AN API TO MANAGE EVOLUTION
We need “microservices” for Data
My Data API
My Raw Data
A DATA ACCESS LAYER FOR LINKEDIN
We built Dali to solve this
Logical Tables + Views
Logical FileSystem
Abstract away under...
Dali: Implementation Details in Context
Dali FileSystem
Processing Engine
(MR, Spark)
Dali Datasets (Tables+Views)
Dataflo...
Simple to Complex
Different Types
Basic Restrictions
Access to dataset based on business need
Privacy by Default
Analysts ...
STEP 1: DATA + METADATA
Solving for Compliant Access
Schema = {
int memberId
String firstName
String lastName
Position[] po...
STEP 2: A MEMBER’S PREFERENCES
Privacy Preferences
A BITMAP DATASET: ONE PER MEMBER
Privacy Preferences
Member Privacy
Preferences
Solving for Compliant Access With Dali
Raw
Dataset
Meta
Data
Member Privacy
Preferences
Dali Reader responsibility:
Given:...
Solving for Compliant Purging With Dali + Gobblin
Raw
Dataset
Meta
Data
Member Privacy
Preferences
Gobblin
Purger
Dali
Rea...
DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The ...
DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The ...
DATA DEMOCRACY + DATA PROTECTION
The Technology Blueprint
WhereHows*
Dali Apache Gobblin*
* Open Source : We can collabora...
Core company value, implemented
by Technology & Process
Privacy By Design
Privacy : Technology + Process
SUSTAINABILITY IS...
Getting Stricter and more complex
Data Protection
Key Takeaways
THE BEAST IS REAL
Stricter regulations in a digital world
...
We’ve established a blueprint to
sustainably address privacy
Learnings at LinkedIn
Key Takeaways
THE BEAST CAN BE TAMED !
...
DATA DEMOCRACY <> DATA PROTECTION
More Data
Discover Data
Easy Access
Less Data
Discover Violations
Restricted Access
The ...
Thank You!
Upcoming SlideShare
Loading in …5
×

Published on

The LinkedIn Privacy Paradox
“On one hand, the company has
500+ million members trusting
the company to protect highly
sensitive data.
On the other hand, one only
joins the largest professional
network on the Internet because
they want to be found !"     
     
Kalinda Raina,
Head of Global Privacy, LinkedIn
MEMBER PRIVACY <> MEMBER DISCOVERY

Published in: Technology
  • I want to quickly tell the world that there is a real on line spell caster that is powerful and genuine, His name is Dr ozalogbo He helped me recently to reunite my relationship with my husband who left me, When i contacted Dr ozalogbo he cast a love spell for me, and my husband who said he doesn't have anything to do with me again called me and started begging me to come back. he is back now with so much love and care. today i am glad to let you all know that this spell caster have the powers to restore broken relationship back. because i am now happy with my husband. To anyone who is reading this article and needs any help, Dr ozalogbo can also offer any type of help like, Curing of all types of Diseases, Court Cases, Pregnancy Spell, Spiritual protection and lot's more. You can contact him Via his email [email protected] or [email protected] call or add him on whatsapp with his phone number +2348162562991
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • jb­e­f­or­e ­i s­Aw t­h­e r­e­c­e­i­pt ­f­or $6821, ­i ­d­i­d ­n­ot ­b­e­l­i­ev­e t­h­At...­my... s­ist­er w­oz ­l­i­k­e r­e­A­l­ly ­m­A­k­i­n­g ­m­o­n­ey ­p­Art t­i­m­e ­At t­h­e­ir ­l­A­pt­o­p.. t­h­er­e s­ist­ers ­n­e­i­g­h­b­our ­h­Az ­d­o­n­e t­h­is ­f­or u­n­d­er ­e­l­ev­e­n ­m­o­nt­hs ­A­n­d ­At ­pr­es­e­nt ­p­A­i­d t­h­e ­l­o­A­ns ­o­n t­h­er­e ­A­p­Art­m­e­nt ­A­n­d ­pur­c­h­As­e­d ­A t­o­p ­o­f t­h­e r­A­n­g­e ­l­otus ­c­Ar­lt­o­n. r­e­A­d ­m­or­e ­At,­ COPY AND PASTE&gt;&gt;&gt;&gt;www.jobsish.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • This is the process I went through to get back my online lover who is now my husband to be. Note: I got him back through Dr. Wakina love spell by virtue of [email protected] HERE IS HOW IT STARTED: I said to Dr. Wakina; Sir, I humbly asked for help in uniting with my online lover, we broke up five months ago. I am writing to you as I have witnessed how your love spell helped my colleague soothe her broken relationship. Two years ago, I met an amazing man online via twitter, we bonded within weeks like we have known each other for long even though we were thousands of miles apart. We often talk on the phone, texting and emailing. When it was our next step to meet in person and see if there was truly the spark of love we were both feeling deeply. I received a breakup text, he explained how it is difficult for him to move on without his ex-wife. I was heartbroken because I thought I have found the man of my dream. Can you help me? HERE IS HOW IT ENDS: Dr. Waking revived my hope when I was told that He is my husband and future but He needs to be untied with his past by breaking the bonds of affliction with his ex. My colleague’s breakthrough brought everlasting joy to me, I could not have heard of love spell or Dr. Wakina. APPRECIATION: Why I am happy to share this breakthrough testimony is because the spell worked according to his will and purpose, he is right here with me happy and loving, we are currently planning on getting married. I will always appreciate the opportunity given to me to reconnect with my lover by Dr. Wakina, I have confirmed that we are meant to be forever.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hello to every one that is reading this testimony. I have been rejected by my husband after three(2) years of marriage just because another woman had a spell on him and he left me and the kids to suffer. one day when i was seaching on the internet i saw a post on how this great spell caster on this Website have help a woman to get back her husband and i gave him a reply to his website solution home spell caster and he told me that a woman had a spell on my husband and he told me that he will help me and after 2 days that i will have my husband back. i believed him and today i am glad to let you all know that this spell caster have the power to bring lovers back. because i am now happy with my husband. Thanks to Dr. Ahmed solution Home.what a great celebration. i will keep on testifying on the internet because Dr. Ahmed is truly a real spell caster. DO YOU NEED HELP THEN CONTACT Dr.Ahmed His email [email protected] You can Call &amp; WhatsApp +27631539694:
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hello, Welcome to MR. Dominic Anthony LOAN SERVICE. email us on [email protected] Are You Desperately in need of a loan help? Have you be denied of a loan from your bank or any Financial Firm? Do you need financial assistance? Do you need a loan to pay off your bills or buy a home? Do you want to have a Business of your own and you need Financial Loan Help? Contact us today for your Financial Loan Help. We are willing to help you out on either Business or Personal Loans. Offer are Available at %2 interest rate. APPLICATION DETAILS First Name:________________________ __ Last Name:_________________________ ___ Gender:_______________________ ________ Marital status:_______________________ Contact Address:______________________ City/Zip code:________________________ Country:_____________________ ________ Date of Birth:________________________ Amount Needed as Loan:________________ Loan Duration:_____________________ ___ Monthly Income/Yearly Income:_________ Occupation:__________________ _______ Purpose for Loan:___________________ House Phone:________________________ ________ Cell Phone:________________________ ________ Have you apply for a loan before? ________ Email: [email protected] Regards, CEO Mr.Dominic Anthony
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
monaliza.kiev.ua

uk surrogacy

ry-diplomer.com

×