Download Accountable systems or how to catch a liar?

Accountable systems
or
how to catch a liar?
Jinyang Li
(with slides from authors of SUNDR and PeerReview)
What we have learnt so far
• Use BFT replication to handle (a few)
malicious servers
• Strengths:
– Transparent (service goes on as if no faults)
– Minimal client modifications
• Weaknesses:
– Breaks down completely if >1/3 servers are bad
Hold (bad) servers accountable
• What is possible when >1/3 servers fail?
– E.g. 1 out of 1 servers fail?
• Let us consider a bad file server
–
–
–
–
–
–
Can it delete a client’s data?
Can it leak the content of a client’s data?
Can it change a client’s data to arbitrary values?
Can it show client A garbage and claim it’s from B?
Can it show client A old data of client B?
Can it show A (op1, op2) and B (op2, op1)?
Hold (bad) servers accountable
• Lessons:
– Cannot prevent a bad server from
misbehaving at least once
– We can do many checks to detect past
misbehaviors!
• Useful?
Fool me once, shame
on you; fool me twice,
QuickTime™ and a
decompressor
shame on …
are needed to see this picture.
Case study I: SUNDR
• What’s SUNDR?
– A (single server) network file system
– Handle potential Byzantine server behavior
– Can run on an un-trusted server
• Useful properties:
– Tamper-evident
• Unauthorized operations will be immediately detected
– Can detect past misbehavior
• If server drops operations, can be caught eventually
Ideal File system semantics
• Represent FS calls as fetch/modify operations
– Fetch: client downloads new data
– Modify: client makes new change visible to others
• Ideal (sequential) consistency:
– A fetch reflects the sequence of modifications that
happen before it
– Impossible when the server is malicious
• A is only aware of B’s latest modification via the server
• Goal:
– get as close to ideal consistency as possible
Strawman File System
A: echo “A was here” >> /share/aaa
userA
A
Modify f1
sig1
B
Fetch f4
sig2
A
Modify f2
sig3
File server
sig1
A
Modify f1
sig1
sig2
sig3
userB
A
Modify f1
sig1
B
Fetch f4
sig2
A
Modify f1
sig1
B
Fetch f4
sig2
B
Fetch f4
sig2
B: cat /share/aaa
A
Modify f2
sig3
B
Fetch f2
sig4
A
Modify f2
sig3
Log specifies the total order
A’s latest log:
LogA
A
Modify f1
sig1
B
Fetch f4
sig2
A
Modify f2
sig3
The total order:
LogA ≤ LogB iff LogA
is prefix of LogB
B’s latest log:
LogB
A
Modify f1
sig1
B
Fetch f4
sig2
A
Modify f2
sig3
B
Fetch f2
sig4
Detecting attacks by the server
A: echo “A was here” >> /share/aaa
A
A
Modify f1
sig1
B
Fetch f4
sig2
A
Modify f2
sig3
File server
A
Modify f1
sig1
B
A
Modify f1
sig1
B
Fetch f4
sig2
A
Modify f1
sig1
B
Fetch f4
sig2
B
Fetch f4
sig2
A
Modify f2
sig3
B: cat /share/aaa (stale result!)
B
Fetch f2
sig3
Detecting attacks by the server
A’s latest log:
LogA
A
Modify f1
sig1
B
Fetch f4
sig2
A
Modify f2
sig3a
sig1
sig2
A’s log and B’s log can
no longer be ordered:
LogA ≤ LogB, LogB ≤ LogA
sig3a
B’s latest log:
LogB
A
Modify f1
sig1
B
Fetch f4
sig2
B
Fetch f2
sig3b
What Strawman has achieved
 High overhead, no concurrency
 Tamper-evident
• A bad server can’t make up ops users didn’t do
 Achieves fork consistency
• A bad server can conceal users’ ops from each
other, but such misbehavior could be detected
later
Fork Consistency: A tale of two worlds
File Server
A’s view
…
B’s view
…
Fork consistency is useful
•
Best possible alternative to sequential consistency
– If server lies only once, it has to lie forever
•
Enable detections of misbehavior:
– users periodically gossip to check violations
– or deploy a trusted online “timestamp” box
SUNDR’s tricks to make
strawman practical
1. Store FS as a hash tree
–
No need to reconstruct FS image from log
2. Use version vector to totally order ops
–
No need to fetch entire log to check for misbehavior
Trick #1: Hash tree
Key property:
h0 verifies the entire
tree of data
h
0
D0
h
1
h
3
D2
D1
h
4
D4
h
2
D5
h
5
h
6
D6
D3
h
7
D7
h
8
D8
h
9
h10
D9
D10
h11
D11
h12
D12
Trick #2: version vector
• Each client keeps track of its version #
• Server stores the (freshest) version vector
– Version vector orders all operations
• E.g. (0, 1, 2) ≤ (1, 1, 2)
• A client remembers the latest version vector
given to it by the server
– If new vector does not succeed old one, detect
order violation!
• E.g. (0,1,2) ≤ (0,2,1)
SUNDR architecture
SUNDR server-side
userA
SUNDR
client
block server
Untrusted
Network
userB
consistency server
SUNDR
client
• block server stores blocks retrievable by content-hash
• consistency server orders all events
SUNDR data structures:
hash tree
• Each file is writable by one user or group
• Each user/group manages its own pool of inodes
– A pool of inodes is represented as a hash tree
Hash files
data1
i-node
data2
Metadata
20-byte File Handle
H(data1)
H(data2)
iblk1
H(iblk1)
H(data3)
H(data4)
•
data3
data4
Blocks are stored and indexed by hash on the block server
Hash a pool of i-nodes
•
Hash all files writable by each user/group
i-table
i-num
20-byte digest
2
20-byte
3
20-byte
4
20-byte
i-node 2
i-node 3
i-node 4
• From this digest, client can retrieve and verify any block of
any file
SUNDR FS
Lookup “/”
digest
Superuser:
SUNDR State
digest
UserA:
digest
UserB:
digest
GroupG:
2
3
4
20-byte
20-byte
20-byte
2
3
4
20-byte
20-byte
20-byte
Lookup “/share”
Fetch “/share/aaa”
…
How to fetch “/share/aaa”?
2
3
4
/:
Dir entry: (share, Superuser, 3)
/share:
Dir entry: (aaa, UserA, 4)
SUNDR data structures:
version vector
• Server orders users’ fetch/modify ops
• Client signs version vector along with digest
• Version vectors will expose ordering failures
Version structure
A-1
B-1
Digest A G - 1
Signature A
A
VSTA
≤
A-1
B-2
Digest B G - 2
Signature B
B
VSTB
• Each user has its own version structure (VST)
• Consistency server keeps latest VSTs of all users
• Clients fetch all other users’ VSTs from server before
each operation and cache them
• We order VSTA ≤ VSTB iff all the version numbers in VSTA
are less than or equal in VSTB
Update VST: An example
A
A-0
DigA
B-0
A
A-1
DigA
B-1
Consistency
Server
A
B
A-0
DigB
B-1
A: echo “A was here”
>> /share/aaa
A
A-1
DigA
B-1
VSTA ≤ VSTB
B
A
A-1
DigA
B-1
B: cat /share/aaa
B
A-1
DigB
B-2
B
A-0
DigB
B-1
B
A-1
DigB
B-2
Detect attacks
A
Consistency Server
A
A-0
DigA
B-0
A
A-1
DigA
B-1
B
A-0
DigA
B-1
A: echo “A was here”
>> /share/aaa
B
A
A-1
B
A-0
DigA
B-1
DigB
B-1
B
A-0
DigB
B-2
≤
A’s latest VST and B’s can
no longer be ordered:
VSTA ≤ VSTB, VSTB ≤ VSTA
A
A-0
DigA
B-0
B: cat /share/aaa (stale!)
B
A-0
DigB
B-2
Support concurrent operations
• Clients may issue operations concurrently
• How does second client know what vector to sign?
– If operations don’t conflict, can just include first user’s
forthcoming version number in VST
– But how do you know if concurrent operations conflict?
• Solution: Pre-declare operations in signed updates
– Server returns latest VSTs and all pending updates,
thereby ordering them before current operation
– User computes new VST including pending updates
– User signs and commits new VST
Concurrent update of VSTs
Consistency Server
A
A
A-0
DigA
B-0
B
update: A-1
B
A-0
DigB
B-1
B
A-1
DigB
B-2
update: B-2
A-1
A
A-1
DigA
B-1
B
A-0
DigB
B-1
A: echo “A was here”
>>/share/aaa
A-1
A
A-1
DigA
B-1
VSTA ≤ VSTB
A-1
A
A-0
DigA
B-0
A-1 B-2
B: cat /share/bbb
B
A-1
DigB
B-2
A-1 B-2
A-1 B-2
SUNDR is practical
12
10
Seconds
8
6
4
2
0
Create (1K)
NFSv2
NFSv3
Read (1K)
Unlink
SUNDR
SUNDR/NVRAM
Case study II:
PeerReview [SOSP07]
Motivations for PeerReview
• Large distributed systems consist of
many nodes
• Some nodes become Byzantine
– Software compromise
– Malicious/careless administrator
• Goal: detect past misbehavior of nodes
– Apply to more general apps than FS
Challenges of general fault
detection
• How to detect faults?
• How to convince others that a node is (not)
faulty?
Overall approach
• Fault := Node deviates from expected
behavior
• Obtain a signature for every action from
each node
• If a node misbehaves, the signature
works a proof-of-misbehavior against the
node
Can we detect all faults?
1001010110
0010110101
0
1100100100
• No
A
– e.g. Faults affecting a node's
internal state
• Detect observable faults
– E.g. bad nodes send a message
that correct nodes would not send
C
Can we always get a proof?
A
• No
– A said it sent X
B said A didn‘t
C: did A send X?
• Generate verifiable evidence:
?
B
I never
received X!
– a proof of misbehavior (A send wrong X)
– a challenge (C asks A to send X again)
• Nodes not answering challenges are suspects
?!
C
I sent X!
PeerReview Overview
• Treat each node as a deterministic state machine
• Nodes sign every output message
• A witness checks that another node outputs correct
messages
– using a reference implementation and signed inputs
PeerReview architecture
A's witnesses
C
– Including all messages
D
E
• Each node has a set of
witnesses, who audit its
log periodically
• If the witnesses detect
misbehavior, they
– generate evidence
– make the evidence avai-lable
to other nodes
A
A's log
• All nodes keep a log of
their inputs & outputs
B
B's log
• Other nodes check evidence, report fault
PeerReview detects tampering
Message
Hash(log)
B
A
ACK
Hash(log)
B's log
H4
Recv(M)
H3
Send(Z)
H2
H1
H0
Recv(Y)
Send(X)
• What if a node modifies its
log entries?
• Log entries form a hash chain
• Signed hash is included with
every message
 Node commits to having
received all prior messages
PeerReview detects inconsistency
• What if a node
"View #1"
"View #2"
H4 '
Not found
H4
H3
H3 '
Read X
H0
Create X
H2
OK
H1
Read Z
OK
– keeps multiple logs?
– forks its log?
• Witness checks if signed
hashes form a single
chain
State machine
PeerReview detects faults
• Witness audits a node:
Module A
Module B
Log
Network
Module A
Module B
Input
Output
=?
if ≠
– Replay inputs on a
reference implementation
of the state machine
– Check outputs against
the log
PeerReview‘s guarantees
• Faults will be detected (eventually)
– If a node if faulty:
• Its witness has a proof of misbehavior
• Its witness generates a challenge that it cannot answer
– If a node is correct
• No proof of misbehavor
• It can answer any challenge
PeerReview applications
• App #1: NFS server
• Tampering with files
• Lost updates
• Metadata corruption
• Incorrect access control
• App #2: Overlay multicast
• Freeloading
• Tampering with content
• App #3: P2P email
• Denial of service
• Dropping emails
PeerReview’s performance penalty
Avg traffic (Kbps/node)
100
80
60
Baseline traffic
40
20
0
Baseline
1
2
3
Number of witnesses
4
5
• Cost increases w/ # of witnesses per
node W
What have we learnt?
• Put constraints on what faulty servers can do
– Clients sign data, bad SUNDR server cannot fake
data
– Clients sign version vector, bad server cannot
hide past inconsistency
• Fault detection
– Need proof of misbehavior (by signing actions)
– Use challenges to distinguish slow nodes apart
from bad ones