Mereview artikel

Salah satu cara memberikan kontribusi kepada komunitas akademik, selain melakukan publikasi, adalah dengan menjadi reviewer. Dengan menjadi reviewer, kita akan terbiasa membaca ‘pikiran’ orang dan dapat belajar banyak.

Saya tidak tahu terjemahan yang enak dari kata ini dalam bahasa Indonesia. Kalau saya terjemahkan sebagai ‘pengkaji’ atau ‘peninjau’, kok terdengar aneh, atau paling tidak terasa asing. Kita gunakan istilah aslinya dalam bahasa Inggris: reviewer.

Ada beberapa hal yang perlu kita perhatikan ketika diminta bantuan untuk mereview.

KENOL, usahakan membaca artikel lebih dari sekali sebelum memberikan komentar. Karena bisa jadi, pertanyaan atau ‘kernyitan dahi’ Anda terjawab setelah membaca artikel kedua kali. 😉

PERTAMA (sengaja saya buat kapital, karena sepanjang ‘jalan’, Anda akan temukan banyak ‘pertama’), pastikan kita menguasai atau paling tidak pernah bersentuhan dengan topik artikel yang akan kita review. Untuk memastikan ini, seringkali dalam beberapa formulir review, kita diminta menuliskan tingkat kepakaran kita dalam topik yang kita review, mulai dari pemula, menengah, sampai dengan mahir/pakar. Apa yang terjadi kalau Anda memaksa? Biasanya hasil review akan terlihat ala kadarnya. Berikut contohnya:

1. Is it valid enough to justify your argument based on apple to non-apple comparison? What I mean is, you were trying to compare between what is practiced in developed to developing countries.
2. Formatting need to be aligned with the proceeding format provided by the organizing committee.
3. English need to be improved.
4. Beware of wording

Hasil review ini saya dapatkan untuk sebuah artikel yang saya kirim ke sebuah konferensi internasional di Indonesia. Nampaknya, panitia agak sembrono tidak menghapus metadata file yang dikirim, sehingga saya tahu siapa reviewernya. Saya terus terang tidak paham yang dimaksudkan reviewer. Bandingkan dengan hasil review yang satu ini.

Clear, succinct abstract. The premise of the paper is clear in the Introduction – it would probably help to explain a little more why we should be interested in the bidrectionality of the relation: yes, it is interesting to know what causes differences in levels of e-government diffusion/use but why – of all the factors which affect that – would be identify level of corruption as potentially relevant? The review of the impact of e-gov on corruption is fine, though the para “Transparency may …” needs to bring in ICTs/e-gov. The review of the opposite relation is weaker, and might be better to start by asking if there is a link between level of corruption (i.e. a broader measure than anti-corruption practices) and level of e-government. I do know one paper on this – Jennifer Bussell’s on “Political Incentives and Policy Outcomes: Who Benefits from Technology Enabled Serivice Centers?” You can find it as part of the ICTD2009 conference proceedings (beware – 27MB file): http://www.ictd2009.org/documents/ICTD2009Proceedings.pdf For Section III, one thing you should just check – did the approach to scoring (i.e. the elements and weights of the score) for both UN and TI indexes stay the same for all five data points? I know the UN one has changed at some point in recent years. I didn’t understand how Figures 2 or 3 could support a claim about the impact of e-gov on anti-corruption: surely these graphs just show correlation – and the not unexpected result that richer countries tend to both have more e-gov and less corruption – they do not show impact; to even suggest that you would need to undertake the lagged analysis. More generally, I would be suspicious about the time lags used. For example, e-government does not instantly have an impact on corruption; it would surely be better to measure the relation of EGDI 2008 to CPI 2010. There is surely no basis to identify the EGDI 2010 to CPI 2010 as potentially causal. And aren’t you confusing statistical tests – claiming that R2 (a measure of correlation) can be used to state how much of e-gov can be explained by anti-corruption: for that latter you’d need to do an analysis of variance test. You also seem to be confusing the fact that – because the correlations you find are essentially similar with the different data sets, somehow that implies a bi-directional causality. It doesn’t. It just shows a correlation. So, overall, this is an intriguing paper and will no doubt stimulate debate, but it seems to have some fundamental statistical problems when it seeks to move beyond correlation. The finding that the correlation between e-gov and anti-corruption is much lower in developed than developing countries is the main credible outcome, and the paper ought to do more to try to explain why that should be – perhaps Bussell’s paper might offer some insights on that.

Terlihat beda bukan? Lagi-lagi karena metadata tidak dihapus, saya mengetahui identitas reviewer. Bahkan saya katakan ke kolega di kampus, “Kalaupun artikel saya ditolak oleh ‘dia’, saya rela.’ Reviewer ini dari University of Manchester dan kalau ada peneliti eGovernment di negara berkembang kok tidak ‘kenal’ dia, nampaknya kebangeten sekali. 😉 Dua hasil review ini menunjukkan bahwa yang pertama tidak mengenal dengan baik topik (paling tidak ini pendapat saya setelah melihat hasil review) dan kedua terlihat menguasai.

KEDUA, selalu perlihatkan sisi positif artikel, sebelum menuliskan kelemahan. Ini masalah etika. Sejelek apapun artikel, pasti ada sisi positifnya. Gunakan bahasa yang santun, sebagaimana Anda ingin diperlakukan. Tidak jarang meskipun sudah menjadi ahli dalam bidang tertentu, kadang reviewer lupa tentang hal ini. Berikut contoh hasil review yang saya terima untuk artikel yang sama. Review ini dari salah satu senior editor.

It was rejected from the Global Dev workshop because it has some fundamental mistakes that cannot be fixed.
1. You cannot conduct an regression on an index which is composed of several measures, some of which you maybe using as independent variables.
2. The e-Government Index is also a ranking composed of three indexes. Corruption index is also a composite measure.
3. The general mathematical formulae proposed at top of page 5 by the researcher are tautological.
This someone who has limited knowledge of statistics and even less knowledge of how to use statistical methods in scientific research.

Apa masalanya? Pertama, dia menuduh dengan serta merta. Padahal artikel yang dia review tidak pernah saya kirimkan ke Global Dev workshop. Kedua, ada banyak madzhab dalam statistika. Madzhab reviewer nampaknya adalah konservatif. Ketiga, banyak penjelasan dan argumen yang tidak diperhatikan. Namun toh, setelah revisi di beberapa bagian (terutama menguatkan argumen dan menyederhanakan analisis statistik), artikel ini sudah diterbitkan di outlet lain. Keempat, dia menyerang personal, bukan argumen.

Meskipun demikian, jika Anda menerima hasil review seperti ini, jangan tersinggung. Sedikit sedih wajar, tapi jangan terlalu lama. Semua penulis ‘seharusnya’ pernah merasakan artikelnya ditolak. Ketika Prof. Geoff Walsham ditanya di sebuah konferensi di Nepal setahun lalu yang saya ikuti, dia mengatakan tidak pernah mengirim artikel yang langsung diterima tanpa revisi. Anda ingin tahu siapa Geoff Walsham? Silakan ketikkan namanya di mesin pencari.

Selalu ambil sisi positif dengan kepala dingin. Mengapa? Pertama, mereka para reviewer adalah calon pembaca artikel Anda. Minimal Anda mengetahui persepsi calon pembaca yang perlu diperhatikan dalam revisi. Karena bisa jadi cara kita menyampaikan argumen kurang jelas, sehingga disalah-fahami. Kedua, mereka melakukan review dengan sukarela dan tidak dibayar. Jadi Anda mendapatkan penilain jujur terhadap artikel Anda dengan gratis. Apa yang bisa Anda harapkan lebih dari itu?

KETIGA, perhatikan kembali hasil review kedua dan ketiga. Apa bedanya? Yang kedua memberikan saran dan yang ketiga tidak sama sekali. Jadi, selalu berikan saran dalam hasil review Anda, dan tidak hanya menunjukkan masalah. Kalau ini saja, nampaknya lebih banyak orang yang bisa melakukan. Apakah saya punya legitimasi untuk itu? Ini bukan masalah legitimasi. Ini masalah memberi saran, memberi jalan keluar. Misalnya, jika Anda menemukan bahwa artikel tidak menggunakan teori. Jangan hanya disalahkan, tetapi sarankan teori apa yang tepat atau mungkin digunakan. Toh keputusan juga ada di penulis, bukan reviewer.

KEEMPAT, jika panitia memberikan rambu-rambu atau kriteria penilaian, pastikan Anda mengikutinya. Sangat mungkin Anda akan menemukan beragam kriteria yang berbeda antara satu konferensi dengan konferensi lainnya, atau antara satu jurnal dengan jurnal lainnya. Ini adalah bagian dari ‘aturan permainan’. Jika Anda mengikuti, insya Allah semuanya akan merasa bahagia.

KELIMA, saat ini jurnal sudah semakin peduli dan meminta penulis memberikan deklarasi terkait dengan sumber pendanaan penelitian atau sejenisnya. Tujuannya satu, supaya penelitian bebas dari pesan sponsor, atau jiak tertulis, pembaca bisa hati-hati dalam membaca artikel dan mengambil kesimpulan. Melakukan review juga sama. Perhatikan kembali hasil review pertama dan kedua. Hasil review kedua mengindikasikan bahwa bahasa Inggris yang digunakan dalam artikel ‘good’, tetapi reviewer pertama menyarankan ‘need proofreading service’. Ternyata, panitia bekerjasama dengan lembaga yang memberikan layanan ‘proofreading’. Meskipun tidak menuduh, namun patut diduga ada kepentingan bisnis yang bermain di sana.

Berikut saya lampirkan dua contoh hasil review yang saya lakukan untuk sebuah konferensi internasional beberapa waktu yang lalu. Ini bukan contoh yang bagus, tapi paling tidak dapat memberikan inspirasi. 😉 Formulir review saya adaptasi dari formulir serupa yang dibuat salah satu guru saya, Prof. Peter Axel Nielsen. Kriteria yang saya masukkan adalah yang diinginkan oleh panitia. Perlu diingat, biasanya review untuk artikel konferensi tidak ‘segalak’ artikel jurnal. Dalam konferensi biasanya ‘single-round review’, sedang dalam jurnal ‘miltiple-round review’, bisa lebih dari tiga kali dan memakan waktu yang lama.

Anda mempunyai pengalaman lain, silakan dibagi! Tambahkan yang KELIMA, KEENAM, dan seterusnya. Insya Allah akan bermanfaat untuk komunitas ilmiah Indonesia.

Ada bagian yang tidak Anda setujui, mari kita diskusikan.

Semoga bermanfaat!

Kristiansand, 21 februari 2012

Lampiran:

Contoh #1:

Conference: …
Paper number: …
Type of the paper: Regular paper
Title of the paper: …

Criteria (Score*)
1. The paper is conceptually significant (7)
2. The paper is practically significant (8)
3. The paper is well-designed (8)
4. The design is well-executed (8)
5. The presentation of ideas is effective (7)
6. Overall, the paper presently makes an important contribution (8)

*Score: 1: significantly disagree; 10: significantly agree

Area of concern (A):
The study investigates a successful eGovernment implementation in the context of a developing country, i.e., V. The focuses are describing the ten-year roadmap of eGovernment implementation in the city of X, identifying its key success factors, and explaining how eGovernment transforms the city.  

Research question (RQ):
RQ is not formulated properly.

Research approach (RA):
Exploratory, case study, interpretive.

Framework (F):
No theories or frameworks adopted. The paper only presents findings from previous studies.

Data collection (DC):
Interviews, secondary data (archives, memos, etc.)

Data analysis (DA):
Grounded interpretive.

Contributions (C):
The paper claims three contributions: disclosure of ten-year implementation, a set of key success factors, and values to eGovernment for city transformation.

Command of literature (related field & theoretical foundation):
The paper needs to bring a framework or a theory in. The literature reviews is not arranged in a thematic way.

Comments and recommended revisions:
This paper discusses an important topic about eGovernment implementation in the context of developing countries, which as reported in the literature experienced a high rate of failure. In general, the paper is easy to follow. The introduction is well written, but giving precise formulation of RQ(s) may be very helpful to guide the reader to follow the logic of the paper.

What I miss from the paper is an inspiring theoretical framework that guides (the data collection and) analysis. In considering the aims of the study (and also the presented findings), in this paper, at least there are three themes that may be taken into consideration in developing the theoretical framework: trajectory of eGovernment implementation, key success factors (mainly on the importance of strategic planning, political leadership, public private partnership, implementation strategy/context-aware IT implementation), and impact of eGovernment.

Once the authors has setup RQ(s) in the outset, the discussion may focus to answer the RQ(s), using the theoretical framework to analyze or explain the findings.

The authors may want to check subheading 4.2.2. Is it 2006-2010 or 2006-2007? Please check also title of the references nos. 9, 13, 19, 22, and 24.

Within an ‘A’ a particular research interest is a ‘RQ’ which will be research using a ‘RA’ in association with a relevant ‘F’. The research is performed with a specific ‘DC’, which may be influenced by ‘F‘ and the empirical data are analyzed with ‘DA‘ in which ‘F’ may be used. The ‘DA’ yields a ‘C,’ which in turn should improve ‘A’ and ‘F’.

Contoh #2:

Conference: …
Paper number: …
Type of the paper: Regular paper
Title of the paper: …

Criteria (Score*)
1. The paper is conceptually significant (6)
2. The paper is practically significant (7)
3. The paper is well-designed (7)
4. The design is well-executed (7)
5. The presentation of ideas is effective (4)
6. Overall, the paper presently makes an important contribution (6)

*Score: 1: significantly disagree; 10: significantly agree

Area of concern (A):
The paper focuses on examining institutional interplay, various institutional logics, and their connection to local improvisation in (an) eGovernment project(s) with special reference to ABC.

The abstract states that the study examines the process and outcomes of *an* eGovernment project in Dubai, but the introduction states *a set of* eGovernment projects. This needs clarification.

Research question (RQ):
RQ is not formulated properly.

Research approach (RA):
Interpretive. But why? Give (an) argument(s).

Framework (F):
Technology Enactment Framework (TEF) introduced by Fountain based on institutional theory.

Data collection (DC):
Interviews (with IT professionals), observation (eight months), secondary data

Data analysis (DA):
Not described.

Contributions (C):
The paper concludes that TEF should be extended.

Command of literature (related field & theoretical foundation):
The paper needs more discussion of existing literature, and needs make theoretical basis clearer.

Comments and recommended revisions:
To me, the logic of the paper is difficult to follow. It will be very helpful if the author would like to make a precise RQ that will guide the storyline of the paper and the reader to understand the paper. The introduction should describe why this topic is interesting and relevant.

The use of TEF should be justified, and it needs more elaboration in the theoretical section. Five main constructs/concepts in TEF should be always taken into account when presenting findings and discussion. It will be nice to have a new figure of the ‘contextualized’ TEF with the main findings presented (in the finding section). Another figure of the extended TEF as recommended by the author will be helpful to understand the contributions of the paper (in the discussion or conclusion section). Also be aware that TEF has been criticized in the literature.

Presentation of the findings should be improved. The description is too general and it needs a more specific examples. Referring to the mains concepts of TEF, the findings should describe what institutional arrangements were in place, what objective technology used, what the organization forms adopted, what various institutional logics emerged, what the observed outcomes, etc. Put them also concisely in the figure of the ‘contextualized’ TEF (see previous point). For example, in the paper, the author states that ‘an alternative bureaucratic logic began to take precedence…’, but what is the alternate logic is not presented explicitly, and what is the ‘older’ logic overtaken by the new one. How this logic leads to (a) certain improvisation(s) is also worth to be explained.

Once the author has setup a RQ in outset, the discussion part may focus to answer the RQ, and the framework (TEF) should guide the discussion.

In considering that the author has spent eight months for the field work, I believe that there must be a lot of information obtained. The problem here is about presenting it in a more logical way to make the paper easier to follow and to be understood.

Overall, the paper needs to be reorganized.

Within an ‘A’ a particular research interest is a ‘RQ’ which will be research using a ‘RA’ in association with a relevant ‘F’. The research is performed with a specific ‘DC’, which may be influenced by ‘F‘ and the empirical data are analyzed with ‘DA‘ in which ‘F’ may be used. The ‘DA’ yields a ‘C,’ which in turn should improve ‘A’ and ‘F’.