Fingerprint Generation for Similar Document Retrieval on the Web

• Álvaro R. Pereira Jr UFMG
• Nivio Ziviani UFMG

Abstract

This paper presents a mechanism for the generation of the “fingerprint” of a Web document. This mechanism is part of a system for detecting and retrieving documents from the Web with a similarity relation to a suspicious document. The process is composed of three stages: a) generation of a fingerprint of the suspicious document, b) gathering candidate documents from the Web and c) comparison of each candidate document and the suspicious document. In the first stage, the fingerprint of the suspicious document is used as its identification. The fingerprint is composed of representative sentences of the document. In the second stage, the sentences composing the fingerprint are used as queries submitted to a search engine. The documents identified by the URLs returned from the search engine are collected to form a set of similarity candidate documents. In the third stage, the candidate documents are “in-place” compared to the suspicious document. The focus of this work is on the generation of the fingerprint of the suspicious document. Experiments were performed using a collection of plagiarized documents constructed specially for this work. For the best fingerprint evaluated, on average 87.06% of the source documents used in the composition of the plagiarized document were retrieved from the Web.

References

Brin, S., Davis, J., and Garcia-Molina, H. (1995). Copy detection mechanisms for digital documents. In ACM SIGMOD Annual Conference, pages 398–409, San Francisco.

Broder, A. (1998). On the resemblance and containment of documents. In Compression and Complexity of Sequences (SEQUENCES’97), pages 21–29. IEEE Computer Society.

Garcia-Molina, H., Gravano, L., and Shivakumar, N. (1996). dscam : Finding document copies across multiple databases. In 4th International Conference on Parallel and Distributed Systems (PDIS’96), Miami Beach.

Garcia-Molina, H., Ketchpel, S. P., and Shivakumar, N. (1998). Safeguarding and charging for information on the internet. In International Conference on Data Engineering (ICDE’98).

Manber, U. (1994). Finding similar files in a large file system. In Proceedings of the USENIX Winter 1994 Technical Conference, pages 1–10, San Fransisco, CA, USA.

Morrison, D. R. (1968). Practical algorithm to retrieve information coded in alphanumeric. ACM, 15(4):514–534.

Pereira-Jr, A. R. (2004). Recuperação de documentos similares na web. Master’s thesis, Departamento de Ciência da Computação da Universidade Federal de Minas Gerais, Belo Horizonte, Brasil.

Pereira-Jr, A. R. and Ziviani, N. (2003). Syntactic similarity of web documents. In First Latin American Web Congress, pages 194–200, Santiago, Chile.

Shivakumar, N. and Garcia-Molina, H. (1995). Scam: A copy detection mechanism for digital documents. In 2nd International Conference in Theory and Practice of Digital Libraries (DL’95), Austin, Texas.

Stricherz, M. (2001). Many teachers ignore cheating, survey finds. Education Week. [link].