[Ltg] HAIL Seminar (9th September): Dr Paul Thomas, CSIRO ICT Centre

[Andrew Lampert]

                      H.A.I.L. Seminar series
                          CSIRO ICT Centre
                   http://www.ict.csiro.au/HAIL/


Title:          Relevance assessment:
                    Are judges exchangeable and does it matter?

Speaker:        Dr Paul Thomas
                Postdoctoral Fellow
                Information Engineering Laboratory
                CSIRO ICT Centre

Date:           Tuesday 9th September 2008 at 11am

Location:       CSIRO ICT Centre,
                Building E6B, Macquarie University.

                See <http://www.ict.csiro.au/HAIL/location.htm> for details.

Video:          We usually stream live video of seminars.

                At the seminar time (see above), point your browser at:
                  <http://webcast.nsw.csiro.au/httpfs/ICT/HailSeminar/live.asx>


Abstract

Reusable test collections for information retrieval rely on relevance judgements, decisions on which documents are good answers to each query in the test. We investigate to what extent people making relevance judgements are interchangeable. Analysis shows low levels of agreement between judges, and we report on experiments to determine if this is sufficient to invalidate the use of a test collection.

We find that both system scores and system rankings are subject to consistent but small differences across different relevance judges. It appears that collections are not completely robust to changes of judge when these judges vary widely in task and topic expertise. This has implications for anyone building a reusable test collection.

(This is joint work with colleagues at NIST, Microsoft, and CWI.)


Short resume

Paul is a postdoctoral fellow at the CSIRO ICT Centre, working largely on problems in distributed and personal information retrieval. He completed his PhD, at the ANU, earlier this year.


----------------------------------------
The HAIL Seminars' URL:
http://www.ict.csiro.au/HAIL/

Contacts:       Andrew Lampert
Address:        CSIRO HAIL Seminars,
                c/o Andrew Lampert,
                Locked Bag 17,
                North Ryde NSW 1670

Phone:  (02) 9325 3100
Email:  Andrew.Lampert at csiro.au


*Administration*
-----------------------------------------------------------------
* To leave the list, send the message "unsubscribe HailSeminars" to
the list server address <majordomo at nsw.cmis.csiro.au>.
The subject line of your e-mail message will be ignored.
==================================================================